You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/03/05 06:48:40 UTC

[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1383 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1383/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC -XX:-UseSuperWord

5 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestDistribDocBasedVersion

Error Message:
ERROR: SolrIndexSearcher opens=13 closes=11

Stack Trace:
java.lang.AssertionError: ERROR: SolrIndexSearcher opens=13 closes=11
	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:418)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:177)
	at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestDistribDocBasedVersion

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.TestDistribDocBasedVersion:     1) Thread[id=10253, name=searcherExecutor-6433-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=10273, name=searcherExecutor-6441-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestDistribDocBasedVersion: 
   1) Thread[id=10253, name=searcherExecutor-6433-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   2) Thread[id=10273, name=searcherExecutor-6441-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestDistribDocBasedVersion

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=10273, name=searcherExecutor-6441-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=10253, name=searcherExecutor-6433-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=10273, name=searcherExecutor-6441-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
   2) Thread[id=10253, name=searcherExecutor-6433-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestDistribDocBasedVersion

Error Message:
Captured an uncaught exception in thread: Thread[id=10295, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestDistribDocBasedVersion]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10295, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestDistribDocBasedVersion]
Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)
	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.lang.AssertionError
	at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:3809)
	at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:3793)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3646)
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestDistribDocBasedVersion

Error Message:
Captured an uncaught exception in thread: Thread[id=10296, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestDistribDocBasedVersion]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10296, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestDistribDocBasedVersion]
Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)
	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.lang.AssertionError
	at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:3809)
	at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:3793)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3646)
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)




Build Log:
[...truncated 10920 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestDistribDocBasedVersion
   [junit4]   2> 3186699 T10169 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 3186699 T10169 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /iekf/
   [junit4]   2> 3186705 T10169 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solrtest-TestDistribDocBasedVersion-1393996570292
   [junit4]   2> 3186706 T10169 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 3186707 T10170 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 3186808 T10169 oasc.ZkTestServer.run start zk server on port:57960
   [junit4]   2> 3186809 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3186817 T10176 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f997402 name:ZooKeeperConnection Watcher:127.0.0.1:57960 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3186820 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3186820 T10169 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 3186831 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3186834 T10178 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b24d71d name:ZooKeeperConnection Watcher:127.0.0.1:57960/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3186835 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3186835 T10169 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 3186842 T10169 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 3186850 T10169 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 3186861 T10169 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 3186868 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-externalversionconstraint.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3186869 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 3186879 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 3186879 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 3186886 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3186886 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3186893 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3186893 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 3186899 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3186900 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 3186906 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3186906 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 3186912 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3186913 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 3186919 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3186920 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3186926 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3186927 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3186934 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3186934 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 3186940 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3186940 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 3186949 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3186952 T10180 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b2ab283 name:ZooKeeperConnection Watcher:127.0.0.1:57960/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3186953 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3187953 T10169 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3187957 T10169 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 3187965 T10169 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:57964
   [junit4]   2> 3187967 T10169 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3187967 T10169 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3187967 T10169 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544
   [junit4]   2> 3187967 T10169 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/'
   [junit4]   2> 3187995 T10169 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/solr.xml
   [junit4]   2> 3188065 T10169 oasc.CoreContainer.<init> New CoreContainer 620248813
   [junit4]   2> 3188065 T10169 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/]
   [junit4]   2> 3188068 T10169 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3188068 T10169 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 3188068 T10169 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3188068 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3188069 T10169 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3188069 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3188069 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3188069 T10169 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3188069 T10169 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3188081 T10169 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3188082 T10169 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3188082 T10169 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3188082 T10169 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57960/solr
   [junit4]   2> 3188083 T10169 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3188084 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3188087 T10191 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e1a9cd0 name:ZooKeeperConnection Watcher:127.0.0.1:57960 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3188088 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3188091 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3188094 T10193 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49008971 name:ZooKeeperConnection Watcher:127.0.0.1:57960/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3188094 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3188097 T10169 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 3188104 T10169 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 3188113 T10169 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 3188117 T10169 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57964_iekf
   [junit4]   2> 3188119 T10169 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57964_iekf
   [junit4]   2> 3188125 T10169 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 3188129 T10169 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 3188137 T10169 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:57964_iekf
   [junit4]   2> 3188137 T10169 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 3188141 T10169 oasc.Overseer.start Overseer (id=91356959230984196-127.0.0.1:57964_iekf-n_0000000000) starting
   [junit4]   2> 3188147 T10169 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 3188158 T10195 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 3188158 T10169 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 3188159 T10195 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 3188162 T10169 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 3188166 T10169 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3188172 T10194 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 3188181 T10196 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 3188181 T10196 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3188183 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3188183 T10196 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3188186 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3188187 T10194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:57964/iekf",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57964_iekf",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 3188187 T10194 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 3188187 T10194 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 3188192 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3188193 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3189185 T10196 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 3189186 T10196 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/collection1
   [junit4]   2> 3189186 T10196 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 3189187 T10196 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3189187 T10196 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 3189189 T10196 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3189190 T10196 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/collection1/'
   [junit4]   2> 3189191 T10196 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/collection1/lib/.svn/' to classloader
   [junit4]   2> 3189191 T10196 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/collection1/lib/classes/' to classloader
   [junit4]   2> 3189191 T10196 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/collection1/lib/README' to classloader
   [junit4]   2> 3189247 T10196 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 3189308 T10196 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3189311 T10196 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3189317 T10196 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 3189796 T10196 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3189799 T10196 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3189800 T10196 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3189806 T10196 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3189851 T10196 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
   [junit4]   2> 3189852 T10196 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/collection1/, dataDir=./org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/control/data/
   [junit4]   2> 3189852 T10196 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@13f9690e
   [junit4]   2> 3189854 T10196 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/control/data
   [junit4]   2> 3189854 T10196 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/control/data/index/
   [junit4]   2> 3189855 T10196 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 3189856 T10196 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/control/data/index
   [junit4]   2> 3189856 T10196 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=47, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3189862 T10196 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/control/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@827c4a3; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 3189862 T10196 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3189866 T10196 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-constraint" (default)
   [junit4]   2> 3189867 T10196 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-constraint" (default)
   [junit4]   2> 3189867 T10196 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-float"
   [junit4]   2> 3189867 T10196 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-float"
   [junit4]   2> 3189868 T10196 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-failhard"
   [junit4]   2> 3189868 T10196 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-failhard"
   [junit4]   2> 3189868 T10196 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 3189868 T10196 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 3189869 T10196 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 3189869 T10196 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 3189871 T10196 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 3189880 T10196 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3189887 T10196 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 3189887 T10196 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 3189888 T10196 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=2, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3189889 T10196 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/control/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@827c4a3; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 3189889 T10196 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3189889 T10196 oass.SolrIndexSearcher.<init> Opening Searcher@1080329d[collection1] main
   [junit4]   2> 3189890 T10197 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1080329d[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 3189892 T10196 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 3189893 T10169 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 3189894 T10169 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3189893 T10200 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:57964/iekf collection:control_collection shard:shard1
   [junit4]   2> 3189896 T10200 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 3189899 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3189902 T10202 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1124926f name:ZooKeeperConnection Watcher:127.0.0.1:57960/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3189903 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3189904 T10169 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3189908 T10169 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3189909 T10200 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 3189912 T10200 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 3189912 T10200 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C10077 name=collection1 org.apache.solr.core.SolrCore@289a1491 url=https://127.0.0.1:57964/iekf/collection1 node=127.0.0.1:57964_iekf C10077_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:57964/iekf, core=collection1, node_name=127.0.0.1:57964_iekf}
   [junit4]   2> 3189912 T10200 C10077 P57964 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:57964/iekf/collection1/
   [junit4]   2> 3189913 T10200 C10077 P57964 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 3189913 T10200 C10077 P57964 oasc.SyncStrategy.syncToMe https://127.0.0.1:57964/iekf/collection1/ has no replicas
   [junit4]   2> 3189913 T10200 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:57964/iekf/collection1/ shard1
   [junit4]   2> 3189913 T10200 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 3189916 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3189925 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3189926 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3189933 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3189933 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3189941 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3189945 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3190049 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3190050 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3190082 T10200 oasc.ZkController.register We are https://127.0.0.1:57964/iekf/collection1/ and leader is https://127.0.0.1:57964/iekf/collection1/
   [junit4]   2> 3190082 T10200 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:57964/iekf
   [junit4]   2> 3190082 T10200 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 3190083 T10200 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 3190083 T10200 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3190085 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3190086 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3190086 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3190089 T10200 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3190090 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3190091 T10194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:57964/iekf",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57964_iekf",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 3190098 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3190203 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3190208 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3190895 T10169 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 3190896 T10169 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3190917 T10169 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 3190918 T10169 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:57968
   [junit4]   2> 3190921 T10169 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3190921 T10169 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3190921 T10169 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495
   [junit4]   2> 3190922 T10169 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/'
   [junit4]   2> 3190946 T10169 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/solr.xml
   [junit4]   2> 3191014 T10169 oasc.CoreContainer.<init> New CoreContainer 511968034
   [junit4]   2> 3191015 T10169 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/]
   [junit4]   2> 3191016 T10169 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3191016 T10169 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 3191017 T10169 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3191017 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3191017 T10169 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3191017 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3191018 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3191018 T10169 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3191018 T10169 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3191030 T10169 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3191030 T10169 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3191031 T10169 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3191031 T10169 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57960/solr
   [junit4]   2> 3191031 T10169 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3191032 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3191036 T10213 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f0ae137 name:ZooKeeperConnection Watcher:127.0.0.1:57960 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3191036 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3191040 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3191043 T10215 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@491f9d86 name:ZooKeeperConnection Watcher:127.0.0.1:57960/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3191043 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3191053 T10169 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3192060 T10169 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57968_iekf
   [junit4]   2> 3192062 T10169 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57968_iekf
   [junit4]   2> 3192068 T10202 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 3192068 T10193 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 3192068 T10215 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 3192084 T10216 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 3192084 T10216 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3192086 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3192086 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3192086 T10216 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3192086 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3192090 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3192091 T10194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:57968/iekf",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57968_iekf",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 3192092 T10194 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 3192092 T10194 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 3192097 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3192099 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3192099 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3192099 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3193088 T10216 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 3193089 T10216 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/collection1
   [junit4]   2> 3193089 T10216 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 3193090 T10216 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3193090 T10216 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 3193093 T10216 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3193093 T10216 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/collection1/'
   [junit4]   2> 3193094 T10216 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/collection1/lib/.svn/' to classloader
   [junit4]   2> 3193095 T10216 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/collection1/lib/classes/' to classloader
   [junit4]   2> 3193095 T10216 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/collection1/lib/README' to classloader
   [junit4]   2> 3193153 T10216 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 3193212 T10216 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3193215 T10216 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3193222 T10216 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 3193733 T10216 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3193736 T10216 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3193738 T10216 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3193743 T10216 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3193788 T10216 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
   [junit4]   2> 3193788 T10216 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/collection1/, dataDir=./org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty1/
   [junit4]   2> 3193788 T10216 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@13f9690e
   [junit4]   2> 3193789 T10216 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty1
   [junit4]   2> 3193789 T10216 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty1/index/
   [junit4]   2> 3193790 T10216 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 3193790 T10216 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty1/index
   [junit4]   2> 3193791 T10216 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=47, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3193794 T10216 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty1/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@55818f9b; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 3193794 T10216 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3193800 T10216 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-constraint" (default)
   [junit4]   2> 3193801 T10216 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-constraint" (default)
   [junit4]   2> 3193801 T10216 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-float"
   [junit4]   2> 3193801 T10216 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-float"
   [junit4]   2> 3193802 T10216 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-failhard"
   [junit4]   2> 3193802 T10216 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-failhard"
   [junit4]   2> 3193802 T10216 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 3193803 T10216 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 3193804 T10216 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 3193805 T10216 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 3193805 T10216 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 3193814 T10216 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3193820 T10216 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 3193821 T10216 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 3193822 T10216 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=2, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3193824 T10216 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty1/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@55818f9b; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 3193824 T10216 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3193824 T10216 oass.SolrIndexSearcher.<init> Opening Searcher@554dfd35[collection1] main
   [junit4]   2> 3193825 T10217 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@554dfd35[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 3193828 T10216 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 3193829 T10169 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 3193830 T10220 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:57968/iekf collection:collection1 shard:shard2
   [junit4]   2> 3193830 T10169 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3193832 T10220 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 3193842 T10220 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 3193846 T10220 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 3193846 T10220 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C10078 name=collection1 org.apache.solr.core.SolrCore@b365cbc url=https://127.0.0.1:57968/iekf/collection1 node=127.0.0.1:57968_iekf C10078_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:57968/iekf, core=collection1, node_name=127.0.0.1:57968_iekf}
   [junit4]   2> 3193846 T10220 C10078 P57968 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:57968/iekf/collection1/
   [junit4]   2> 3193846 T10220 C10078 P57968 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 3193846 T10220 C10078 P57968 oasc.SyncStrategy.syncToMe https://127.0.0.1:57968/iekf/collection1/ has no replicas
   [junit4]   2> 3193847 T10220 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:57968/iekf/collection1/ shard2
   [junit4]   2> 3193847 T10220 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 3193850 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3193855 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3193861 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3193864 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3193864 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3193868 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3193874 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3193880 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3193986 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3193986 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3193986 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3194022 T10220 oasc.ZkController.register We are https://127.0.0.1:57968/iekf/collection1/ and leader is https://127.0.0.1:57968/iekf/collection1/
   [junit4]   2> 3194023 T10220 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:57968/iekf
   [junit4]   2> 3194023 T10220 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 3194023 T10220 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 3194023 T10220 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3194025 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3194025 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3194025 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3194026 T10220 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3194029 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3194031 T10194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:57968/iekf",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57968_iekf",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 3194036 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3194140 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3194141 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3194140 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3194886 T10169 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 3194886 T10169 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3194894 T10169 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 3194897 T10169 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:57971
   [junit4]   2> 3194900 T10169 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3194900 T10169 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3194901 T10169 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421
   [junit4]   2> 3194901 T10169 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/'
   [junit4]   2> 3194940 T10169 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/solr.xml
   [junit4]   2> 3195017 T10169 oasc.CoreContainer.<init> New CoreContainer 1957057479
   [junit4]   2> 3195017 T10169 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/]
   [junit4]   2> 3195019 T10169 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3195019 T10169 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 3195020 T10169 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3195020 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3195021 T10169 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3195021 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3195021 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3195021 T10169 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3195021 T10169 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3195036 T10169 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3195036 T10169 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3195036 T10169 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3195037 T10169 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57960/solr
   [junit4]   2> 3195037 T10169 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3195038 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3195045 T10231 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45967778 name:ZooKeeperConnection Watcher:127.0.0.1:57960 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3195045 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3195048 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3195053 T10233 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33135df5 name:ZooKeeperConnection Watcher:127.0.0.1:57960/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3195053 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3195066 T10169 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3196074 T10169 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57971_iekf
   [junit4]   2> 3196076 T10169 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57971_iekf
   [junit4]   2> 3196083 T10193 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 3196083 T10233 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 3196084 T10215 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 3196083 T10202 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 3196101 T10234 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 3196101 T10234 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3196104 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3196104 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3196105 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3196104 T10234 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3196109 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3196110 T10194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:57971/iekf",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57971_iekf",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 3196111 T10194 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 3196111 T10194 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 3196119 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3196120 T10233 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3196121 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3196121 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3196121 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3197107 T10234 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 3197107 T10234 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/collection1
   [junit4]   2> 3197107 T10234 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 3197109 T10234 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3197109 T10234 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 3197111 T10234 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3197111 T10234 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/collection1/'
   [junit4]   2> 3197113 T10234 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/collection1/lib/.svn/' to classloader
   [junit4]   2> 3197113 T10234 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/collection1/lib/classes/' to classloader
   [junit4]   2> 3197113 T10234 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/collection1/lib/README' to classloader
   [junit4]   2> 3197173 T10234 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 3197236 T10234 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3197239 T10234 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3197245 T10234 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 3197730 T10234 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3197734 T10234 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3197736 T10234 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3197742 T10234 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3197785 T10234 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
   [junit4]   2> 3197786 T10234 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/collection1/, dataDir=./org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty2/
   [junit4]   2> 3197786 T10234 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@13f9690e
   [junit4]   2> 3197788 T10234 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty2
   [junit4]   2> 3197788 T10234 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty2/index/
   [junit4]   2> 3197789 T10234 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 3197790 T10234 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty2/index
   [junit4]   2> 3197790 T10234 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=47, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3197793 T10234 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty2/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@20e77e9a; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 3197794 T10234 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3197798 T10234 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-constraint" (default)
   [junit4]   2> 3197799 T10234 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-constraint" (default)
   [junit4]   2> 3197799 T10234 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-float"
   [junit4]   2> 3197800 T10234 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-float"
   [junit4]   2> 3197800 T10234 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-failhard"
   [junit4]   2> 3197800 T10234 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-failhard"
   [junit4]   2> 3197800 T10234 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 3197801 T10234 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 3197801 T10234 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 3197802 T10234 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 3197804 T10234 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 3197814 T10234 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3197821 T10234 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 3197821 T10234 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 3197822 T10234 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=2, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 3197823 T10234 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty2/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@20e77e9a; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 3197823 T10234 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3197823 T10234 oass.SolrIndexSearcher.<init> Opening Searcher@22c9b206[collection1] main
   [junit4]   2> 3197824 T10235 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@22c9b206[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 3197826 T10234 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 3197828 T10169 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 3197828 T10238 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:57971/iekf collection:collection1 shard:shard1
   [junit4]   2> 3197828 T10169 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3197830 T10238 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3197838 T10238 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 3197841 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3197843 T10238 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 3197843 T10238 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C10079 name=collection1 org.apache.solr.core.SolrCore@6c0955af url=https://127.0.0.1:57971/iekf/collection1 node=127.0.0.1:57971_iekf C10079_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:57971/iekf, core=collection1, node_name=127.0.0.1:57971_iekf}
   [junit4]   2> 3197843 T10238 C10079 P57971 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:57971/iekf/collection1/
   [junit4]   2> 3197844 T10238 C10079 P57971 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 3197844 T10238 C10079 P57971 oasc.SyncStrategy.syncToMe https://127.0.0.1:57971/iekf/collection1/ has no replicas
   [junit4]   2> 3197844 T10238 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:57971/iekf/collection1/ shard1
   [junit4]   2> 3197844 T10238 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 3197848 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3197855 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3197859 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3197861 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3197861 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3197861 T10233 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3197868 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3197873 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3197978 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3197979 T10233 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3197979 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3197980 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3198021 T10238 oasc.ZkController.register We are https://127.0.0.1:57971/iekf/collection1/ and leader is https://127.0.0.1:57971/iekf/collection1/
   [junit4]   2> 3198022 T10238 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:57971/iekf
   [junit4]   2> 3198023 T10238 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 3198023 T10238 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 3198023 T10238 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3198025 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3198025 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3198025 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3198027 T10238 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3198029 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3198030 T10194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:57971/iekf",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57971_iekf",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 3198038 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3198149 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3198150 T10233 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3198151 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3198152 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3198958 T10169 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 3198959 T10169 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3198964 T10169 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 3198965 T10169 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:57974
   [junit4]   2> 3198970 T10169 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3198970 T10169 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3198971 T10169 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty3-1393996581418
   [junit4]   2> 3198971 T10169 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty3-1393996581418/'
   [junit4]   2> 3198998 T10169 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty3-1393996581418/solr.xml
   [junit4]   2> 3199083 T10169 oasc.CoreContainer.<init> New CoreContainer 410602430
   [junit4]   2> 3199084 T10169 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty3-1393996581418/]
   [junit4]   2> 3199086 T10169 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3199086 T10169 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 3199087 T10169 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3199087 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3199087 T10169 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3199087 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3199088 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3199088 T10169 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3199088 T10169 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3199101 T10169 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3199101 T10169 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3199102 T10169 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3199102 T10169 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57960/solr
   [junit4]   2> 3199102 T10169 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3199

[...truncated too long message...]

ConfigTest, TestQuerySenderListener, AlternateDirectoryTest, TestQuerySenderNoQuery, ResponseLogComponentTest, SolrIndexConfigTest, BadComponentTest, TestMergePolicyConfig, TestBinaryField, MinimalSchemaTest, TestElisionMultitermQuery, TestConfig, OutputWriterTest, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestSolrCoreProperties, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, TestPhraseSuggestions, TestCharFilters, TestCodecSupport, TestXIncludeConfig, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, LegacyHTMLStripCharFilterTest, TestJmxMonitoredMap, TimeZoneUtilsTest, ScriptEngineTest, PluginInfoTest, DateMathParserTest, PreAnalyzedFieldTest, TestSystemIdResolver, TestSuggestSpellingConverter, DateFieldTest, TestSolrJ, TestUtils, TestDocumentBuilder, ZkNodePropsTest, SliceStateTest, SystemInfoHandlerTest, UUIDFieldTest, CircularListTest, CursorPagingTest, NoFacetTest, AbstractAnalyticsFacetTest, FieldFacetExtrasTest, RangeFacetTest, FunctionTest, AliasIntegrationTest, AssignTest, CustomCollectionTest, DeleteInactiveReplicaTest, DistribCursorPagingTest, MigrateRouteKeyTest, RemoteQueryErrorTest, ShardSplitTest, SolrXmlInZkTest, TestDistribDocBasedVersion]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDistribDocBasedVersion -Dtests.seed=1249227945045A2E -Dtests.slow=true -Dtests.locale=ko_KR -Dtests.timezone=America/Monterrey -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s | TestDistribDocBasedVersion (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrIndexSearcher opens=13 closes=11
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:418)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:177)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestDistribDocBasedVersion: 
   [junit4]    >    1) Thread[id=10253, name=searcherExecutor-6433-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    2) Thread[id=10273, name=searcherExecutor-6441-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=10273, name=searcherExecutor-6441-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    >    2) Thread[id=10253, name=searcherExecutor-6433-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:744)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10295, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestDistribDocBasedVersion]
   [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
   [junit4]    > Caused by: java.lang.AssertionError
   [junit4]    > 	at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:3809)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:3793)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3646)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10296, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestDistribDocBasedVersion]
   [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
   [junit4]    > Caused by: java.lang.AssertionError
   [junit4]    > 	at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:3809)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:3793)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3646)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
   [junit4] Completed in 169.57s, 1 test, 1 failure, 4 errors <<< FAILURES!

[...truncated 135 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:465: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:445: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:490: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1274: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:907: There were test failures: 377 suites, 1616 tests, 5 suite-level errors, 241 ignored (19 assumptions)

Total time: 85 minutes 54 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC -XX:-UseSuperWord
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1383 - Failure!

Posted by Chris Hostetter <ho...@apache.org>.
This is SOLR-5628 ... i still don't know/understand the root cause of the 
deleting files too early problem in hte cloud framework, but i've 
committed a workarround in this particular test since it seems to be the 
only problematic one for now (see issue description for why)

https://issues.apache.org/jira/browse/SOLR-5628


: Date: Wed, 5 Mar 2014 05:48:40 +0000 (UTC)
: From: Policeman Jenkins Server <je...@thetaphi.de>
: To: dev@lucene.apache.org, hossman@apache.org, markrmiller@apache.org
: Subject: [JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1383 -
:     Failure!
: 
: Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1383/
: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC -XX:-UseSuperWord
: 
: 5 tests failed.
: FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestDistribDocBasedVersion
: 
: Error Message:
: ERROR: SolrIndexSearcher opens=13 closes=11
: 
: Stack Trace:
: java.lang.AssertionError: ERROR: SolrIndexSearcher opens=13 closes=11
: 	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)
: 	at org.junit.Assert.fail(Assert.java:93)
: 	at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:418)
: 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:177)
: 	at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
: 	at java.lang.reflect.Method.invoke(Method.java:606)
: 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
: 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
: 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
: 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
: 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
: 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
: 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
: 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
: 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
: 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
: 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
: 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
: 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
: 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
: 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
: 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
: 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
: 	at java.lang.Thread.run(Thread.java:744)
: 
: 
: FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestDistribDocBasedVersion
: 
: Error Message:
: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestDistribDocBasedVersion:     1) Thread[id=10253, name=searcherExecutor-6433-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=10273, name=searcherExecutor-6441-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]         at sun.misc.Unsa
 fe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)
: 
: Stack Trace:
: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestDistribDocBasedVersion: 
:    1) Thread[id=10253, name=searcherExecutor-6433-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]
:         at sun.misc.Unsafe.park(Native Method)
:         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
:         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
:         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
:         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
:         at java.lang.Thread.run(Thread.java:744)
:    2) Thread[id=10273, name=searcherExecutor-6441-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]
:         at sun.misc.Unsafe.park(Native Method)
:         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
:         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
:         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
:         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
:         at java.lang.Thread.run(Thread.java:744)
: 	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)
: 
: 
: FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestDistribDocBasedVersion
: 
: Error Message:
: There are still zombie threads that couldn't be terminated:    1) Thread[id=10273, name=searcherExecutor-6441-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)    2) Thread[id=10253, name=searcherExecutor-6433-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]         at sun.misc.Unsafe.park(Native Method)      
    at java.util.concurrent..locks.LockSupport.park(LockSupport.java:186)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:744)
: 
: Stack Trace:
: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
:    1) Thread[id=10273, name=searcherExecutor-6441-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]
:         at sun.misc.Unsafe.park(Native Method)
:         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
:         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
:         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
:         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
:         at java.lang.Thread.run(Thread.java:744)
:    2) Thread[id=10253, name=searcherExecutor-6433-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]
:         at sun.misc.Unsafe.park(Native Method)
:         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
:         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
:         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
:         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
:         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
:         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
:         at java.lang.Thread.run(Thread.java:744)
: 	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)
: 
: 
: FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestDistribDocBasedVersion
: 
: Error Message:
: Captured an uncaught exception in thread: Thread[id=10295, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestDistribDocBasedVersion]
: 
: Stack Trace:
: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10295, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestDistribDocBasedVersion]
: Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.AssertionError
: 	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)
: 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
: 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
: Caused by: java.lang.AssertionError
: 	at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:3809)
: 	at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:3793)
: 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3646)
: 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
: 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
: 
: 
: FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestDistribDocBasedVersion
: 
: Error Message:
: Captured an uncaught exception in thread: Thread[id=10296, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestDistribDocBasedVersion]
: 
: Stack Trace:
: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10296, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestDistribDocBasedVersion]
: Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.AssertionError
: 	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)
: 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
: 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
: Caused by: java.lang.AssertionError
: 	at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:3809)
: 	at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:3793)
: 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3646)
: 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
: 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
: 
: 
: 
: 
: Build Log:
: [...truncated 10920 lines...]
:    [junit4] Suite: org.apache.solr.cloud.TestDistribDocBasedVersion
:    [junit4]   2> 3186699 T10169 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
:    [junit4]   2> 3186699 T10169 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /iekf/
:    [junit4]   2> 3186705 T10169 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
:    [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solrtest-TestDistribDocBasedVersion-1393996570292
:    [junit4]   2> 3186706 T10169 oasc.ZkTestServer.run STARTING ZK TEST SERVER
:    [junit4]   2> 3186707 T10170 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
:    [junit4]   2> 3186808 T10169 oasc.ZkTestServer.run start zk server on port:57960
:    [junit4]   2> 3186809 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
:    [junit4]   2> 3186817 T10176 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f997402 name:ZooKeeperConnection Watcher:127.0.0.1:57960 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
:    [junit4]   2> 3186820 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
:    [junit4]   2> 3186820 T10169 oascc.SolrZkClient.makePath makePath: /solr
:    [junit4]   2> 3186831 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
:    [junit4]   2> 3186834 T10178 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b24d71d name:ZooKeeperConnection Watcher:127.0.0.1:57960/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
:    [junit4]   2> 3186835 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
:    [junit4]   2> 3186835 T10169 oascc.SolrZkClient.makePath makePath: /collections/collection1
:    [junit4]   2> 3186842 T10169 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
:    [junit4]   2> 3186850 T10169 oascc.SolrZkClient.makePath makePath: /collections/control_collection
:    [junit4]   2> 3186861 T10169 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
:    [junit4]   2> 3186868 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-externalversionconstraint.xml to /configs/conf1/solrconfig.xml
:    [junit4]   2> 3186869 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
:    [junit4]   2> 3186879 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
:    [junit4]   2> 3186879 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
:    [junit4]   2> 3186886 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
:    [junit4]   2> 3186886 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
:    [junit4]   2> 3186893 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
:    [junit4]   2> 3186893 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
:    [junit4]   2> 3186899 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
:    [junit4]   2> 3186900 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
:    [junit4]   2> 3186906 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
:    [junit4]   2> 3186906 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
:    [junit4]   2> 3186912 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
:    [junit4]   2> 3186913 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
:    [junit4]   2> 3186919 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
:    [junit4]   2> 3186920 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
:    [junit4]   2> 3186926 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
:    [junit4]   2> 3186927 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
:    [junit4]   2> 3186934 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
:    [junit4]   2> 3186934 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
:    [junit4]   2> 3186940 T10169 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
:    [junit4]   2> 3186940 T10169 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
:    [junit4]   2> 3186949 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
:    [junit4]   2> 3186952 T10180 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b2ab283 name:ZooKeeperConnection Watcher:127.0.0.1:57960/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
:    [junit4]   2> 3186953 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
:    [junit4]   2> 3187953 T10169 oejs.Server.doStart jetty-8.1.10.v20130312
:    [junit4]   2> 3187957 T10169 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
:    [junit4]   2> 3187965 T10169 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:57964
:    [junit4]   2> 3187967 T10169 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
:    [junit4]   2> 3187967 T10169 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
:    [junit4]   2> 3187967 T10169 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544
:    [junit4]   2> 3187967 T10169 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/'
:    [junit4]   2> 3187995 T10169 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/solr.xml
:    [junit4]   2> 3188065 T10169 oasc.CoreContainer.<init> New CoreContainer 620248813
:    [junit4]   2> 3188065 T10169 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/]
:    [junit4]   2> 3188068 T10169 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
:    [junit4]   2> 3188068 T10169 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
:    [junit4]   2> 3188068 T10169 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
:    [junit4]   2> 3188068 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
:    [junit4]   2> 3188069 T10169 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
:    [junit4]   2> 3188069 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
:    [junit4]   2> 3188069 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
:    [junit4]   2> 3188069 T10169 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
:    [junit4]   2> 3188069 T10169 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
:    [junit4]   2> 3188081 T10169 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
:    [junit4]   2> 3188082 T10169 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
:    [junit4]   2> 3188082 T10169 oasc.CoreContainer.load Host Name: 127.0.0.1
:    [junit4]   2> 3188082 T10169 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57960/solr
:    [junit4]   2> 3188083 T10169 oasc.ZkController.checkChrootPath zkHost includes chroot
:    [junit4]   2> 3188084 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
:    [junit4]   2> 3188087 T10191 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e1a9cd0 name:ZooKeeperConnection Watcher:127.0.0.1:57960 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
:    [junit4]   2> 3188088 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
:    [junit4]   2> 3188091 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
:    [junit4]   2> 3188094 T10193 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49008971 name:ZooKeeperConnection Watcher:127.0.0.1:57960/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
:    [junit4]   2> 3188094 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
:    [junit4]   2> 3188097 T10169 oascc.SolrZkClient.makePath makePath: /overseer/queue
:    [junit4]   2> 3188104 T10169 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
:    [junit4]   2> 3188113 T10169 oascc.SolrZkClient.makePath makePath: /live_nodes
:    [junit4]   2> 3188117 T10169 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57964_iekf
:    [junit4]   2> 3188119 T10169 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57964_iekf
:    [junit4]   2> 3188125 T10169 oascc.SolrZkClient.makePath makePath: /overseer_elect
:    [junit4]   2> 3188129 T10169 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
:    [junit4]   2> 3188137 T10169 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:57964_iekf
:    [junit4]   2> 3188137 T10169 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
:    [junit4]   2> 3188141 T10169 oasc.Overseer.start Overseer (id=91356959230984196-127.0.0.1:57964_iekf-n_0000000000) starting
:    [junit4]   2> 3188147 T10169 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
:    [junit4]   2> 3188158 T10195 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
:    [junit4]   2> 3188158 T10169 oascc.SolrZkClient.makePath makePath: /clusterstate.json
:    [junit4]   2> 3188159 T10195 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
:    [junit4]   2> 3188162 T10169 oascc.SolrZkClient.makePath makePath: /aliases.json
:    [junit4]   2> 3188166 T10169 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
:    [junit4]   2> 3188172 T10194 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
:    [junit4]   2> 3188181 T10196 oasc.ZkController.publish publishing core=collection1 state=down
:    [junit4]   2> 3188181 T10196 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
:    [junit4]   2> 3188183 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3188183 T10196 oasc.ZkController.waitForCoreNodeName look for our core node name
:    [junit4]   2> 3188186 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
:    [junit4]   2> 3188187 T10194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
:    [junit4]   2> 	  "operation":"state",
:    [junit4]   2> 	  "state":"down",
:    [junit4]   2> 	  "base_url":"https://127.0.0.1:57964/iekf",
:    [junit4]   2> 	  "core":"collection1",
:    [junit4]   2> 	  "roles":null,
:    [junit4]   2> 	  "node_name":"127.0.0.1:57964_iekf",
:    [junit4]   2> 	  "shard":null,
:    [junit4]   2> 	  "collection":"control_collection",
:    [junit4]   2> 	  "numShards":"1",
:    [junit4]   2> 	  "core_node_name":null}
:    [junit4]   2> 3188187 T10194 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
:    [junit4]   2> 3188187 T10194 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
:    [junit4]   2> 3188192 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3188193 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
:    [junit4]   2> 3189185 T10196 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
:    [junit4]   2> 3189186 T10196 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/collection1
:    [junit4]   2> 3189186 T10196 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
:    [junit4]   2> 3189187 T10196 oasc.ZkController.createCollectionZkNode Collection zkNode exists
:    [junit4]   2> 3189187 T10196 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
:    [junit4]   2> 3189189 T10196 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
:    [junit4]   2> 3189190 T10196 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/collection1/'
:    [junit4]   2> 3189191 T10196 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/collection1/lib/.svn/' to classloader
:    [junit4]   2> 3189191 T10196 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/collection1/lib/classes/' to classloader
:    [junit4]   2> 3189191 T10196 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/collection1/lib/README' to classloader
:    [junit4]   2> 3189247 T10196 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
:    [junit4]   2> 3189308 T10196 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
:    [junit4]   2> 3189311 T10196 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
:    [junit4]   2> 3189317 T10196 oass.IndexSchema.readSchema [collection1] Schema name=test
:    [junit4]   2> 3189796 T10196 oass.IndexSchema.readSchema default search field in schema is text
:    [junit4]   2> 3189799 T10196 oass.IndexSchema.readSchema unique key field: id
:    [junit4]   2> 3189800 T10196 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
:    [junit4]   2> 3189806 T10196 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
:    [junit4]   2> 3189851 T10196 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
:    [junit4]   2> 3189852 T10196 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.TestDistribDocBasedVersion-controljetty-1393996570544/collection1/, dataDir=./org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/control/data/
:    [junit4]   2> 3189852 T10196 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@13f9690e
:    [junit4]   2> 3189854 T10196 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/control/data
:    [junit4]   2> 3189854 T10196 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/control/data/index/
:    [junit4]   2> 3189855 T10196 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/control/data/index' doesn't exist. Creating new index...
:    [junit4]   2> 3189856 T10196 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/control/data/index
:    [junit4]   2> 3189856 T10196 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=47, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
:    [junit4]   2> 3189862 T10196 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
:    [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/control/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@827c4a3; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
:    [junit4]   2> 3189862 T10196 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
:    [junit4]   2> 3189866 T10196 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-constraint" (default)
:    [junit4]   2> 3189867 T10196 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-constraint" (default)
:    [junit4]   2> 3189867 T10196 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-float"
:    [junit4]   2> 3189867 T10196 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-float"
:    [junit4]   2> 3189868 T10196 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-failhard"
:    [junit4]   2> 3189868 T10196 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-failhard"
:    [junit4]   2> 3189868 T10196 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
:    [junit4]   2> 3189868 T10196 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
:    [junit4]   2> 3189869 T10196 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
:    [junit4]   2> 3189869 T10196 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
:    [junit4]   2> 3189871 T10196 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
:    [junit4]   2> 3189880 T10196 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
:    [junit4]   2> 3189887 T10196 oasu.CommitTracker.<init> Hard AutoCommit: disabled
:    [junit4]   2> 3189887 T10196 oasu.CommitTracker.<init> Soft AutoCommit: disabled
:    [junit4]   2> 3189888 T10196 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=2, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
:    [junit4]   2> 3189889 T10196 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
:    [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/control/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@827c4a3; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
:    [junit4]   2> 3189889 T10196 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
:    [junit4]   2> 3189889 T10196 oass.SolrIndexSearcher.<init> Opening Searcher@1080329d[collection1] main
:    [junit4]   2> 3189890 T10197 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1080329d[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
:    [junit4]   2> 3189892 T10196 oasc.CoreContainer.registerCore registering core: collection1
:    [junit4]   2> 3189893 T10169 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
:    [junit4]   2> 3189894 T10169 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
:    [junit4]   2> 3189893 T10200 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:57964/iekf collection:control_collection shard:shard1
:    [junit4]   2> 3189896 T10200 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
:    [junit4]   2> 3189899 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
:    [junit4]   2> 3189902 T10202 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1124926f name:ZooKeeperConnection Watcher:127.0.0.1:57960/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
:    [junit4]   2> 3189903 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
:    [junit4]   2> 3189904 T10169 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
:    [junit4]   2> 3189908 T10169 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
:    [junit4]   2> 3189909 T10200 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
:    [junit4]   2> 3189912 T10200 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
:    [junit4]   2> 3189912 T10200 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
:    [junit4]   2> ASYNC  NEW_CORE C10077 name=collection1 org.apache.solr.core.SolrCore@289a1491 url=https://127.0.0.1:57964/iekf/collection1 node=127.0.0.1:57964_iekf C10077_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:57964/iekf, core=collection1, node_name=127.0.0.1:57964_iekf}
:    [junit4]   2> 3189912 T10200 C10077 P57964 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:57964/iekf/collection1/
:    [junit4]   2> 3189913 T10200 C10077 P57964 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
:    [junit4]   2> 3189913 T10200 C10077 P57964 oasc.SyncStrategy.syncToMe https://127.0.0.1:57964/iekf/collection1/ has no replicas
:    [junit4]   2> 3189913 T10200 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:57964/iekf/collection1/ shard1
:    [junit4]   2> 3189913 T10200 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
:    [junit4]   2> 3189916 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3189925 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3189926 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
:    [junit4]   2> 3189933 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
:    [junit4]   2> 3189933 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
:    [junit4]   2> 3189941 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
:    [junit4]   2> 3189945 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3190049 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
:    [junit4]   2> 3190050 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
:    [junit4]   2> 3190082 T10200 oasc.ZkController.register We are https://127.0.0.1:57964/iekf/collection1/ and leader is https://127.0.0.1:57964/iekf/collection1/
:    [junit4]   2> 3190082 T10200 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:57964/iekf
:    [junit4]   2> 3190082 T10200 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
:    [junit4]   2> 3190083 T10200 oasc.ZkController.publish publishing core=collection1 state=active
:    [junit4]   2> 3190083 T10200 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
:    [junit4]   2> 3190085 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3190086 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3190086 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3190089 T10200 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
:    [junit4]   2> 3190090 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
:    [junit4]   2> 3190091 T10194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
:    [junit4]   2> 	  "operation":"state",
:    [junit4]   2> 	  "state":"active",
:    [junit4]   2> 	  "base_url":"https://127.0.0.1:57964/iekf",
:    [junit4]   2> 	  "core":"collection1",
:    [junit4]   2> 	  "roles":null,
:    [junit4]   2> 	  "node_name":"127.0.0.1:57964_iekf",
:    [junit4]   2> 	  "shard":"shard1",
:    [junit4]   2> 	  "collection":"control_collection",
:    [junit4]   2> 	  "numShards":"2",
:    [junit4]   2> 	  "core_node_name":"core_node1"}
:    [junit4]   2> 3190098 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3190203 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
:    [junit4]   2> 3190208 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
:    [junit4]   2> 3190895 T10169 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
:    [junit4]   2> 3190896 T10169 oejs.Server.doStart jetty-8.1.10.v20130312
:    [junit4]   2> 3190917 T10169 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
:    [junit4]   2> 3190918 T10169 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:57968
:    [junit4]   2> 3190921 T10169 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
:    [junit4]   2> 3190921 T10169 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
:    [junit4]   2> 3190921 T10169 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495
:    [junit4]   2> 3190922 T10169 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/'
:    [junit4]   2> 3190946 T10169 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/solr.xml
:    [junit4]   2> 3191014 T10169 oasc.CoreContainer.<init> New CoreContainer 511968034
:    [junit4]   2> 3191015 T10169 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/]
:    [junit4]   2> 3191016 T10169 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
:    [junit4]   2> 3191016 T10169 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
:    [junit4]   2> 3191017 T10169 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
:    [junit4]   2> 3191017 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
:    [junit4]   2> 3191017 T10169 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
:    [junit4]   2> 3191017 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
:    [junit4]   2> 3191018 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
:    [junit4]   2> 3191018 T10169 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
:    [junit4]   2> 3191018 T10169 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
:    [junit4]   2> 3191030 T10169 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
:    [junit4]   2> 3191030 T10169 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
:    [junit4]   2> 3191031 T10169 oasc.CoreContainer.load Host Name: 127.0.0.1
:    [junit4]   2> 3191031 T10169 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57960/solr
:    [junit4]   2> 3191031 T10169 oasc.ZkController.checkChrootPath zkHost includes chroot
:    [junit4]   2> 3191032 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
:    [junit4]   2> 3191036 T10213 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f0ae137 name:ZooKeeperConnection Watcher:127.0.0.1:57960 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
:    [junit4]   2> 3191036 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
:    [junit4]   2> 3191040 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
:    [junit4]   2> 3191043 T10215 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@491f9d86 name:ZooKeeperConnection Watcher:127.0.0.1:57960/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
:    [junit4]   2> 3191043 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
:    [junit4]   2> 3191053 T10169 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
:    [junit4]   2> 3192060 T10169 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57968_iekf
:    [junit4]   2> 3192062 T10169 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57968_iekf
:    [junit4]   2> 3192068 T10202 oascc.ZkStateReader$3.process Updating live nodes... (2)
:    [junit4]   2> 3192068 T10193 oascc.ZkStateReader$3.process Updating live nodes... (2)
:    [junit4]   2> 3192068 T10215 oascc.ZkStateReader$3.process Updating live nodes... (2)
:    [junit4]   2> 3192084 T10216 oasc.ZkController.publish publishing core=collection1 state=down
:    [junit4]   2> 3192084 T10216 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
:    [junit4]   2> 3192086 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3192086 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3192086 T10216 oasc.ZkController.waitForCoreNodeName look for our core node name
:    [junit4]   2> 3192086 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3192090 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
:    [junit4]   2> 3192091 T10194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
:    [junit4]   2> 	  "operation":"state",
:    [junit4]   2> 	  "state":"down",
:    [junit4]   2> 	  "base_url":"https://127.0.0.1:57968/iekf",
:    [junit4]   2> 	  "core":"collection1",
:    [junit4]   2> 	  "roles":null,
:    [junit4]   2> 	  "node_name":"127.0.0.1:57968_iekf",
:    [junit4]   2> 	  "shard":null,
:    [junit4]   2> 	  "collection":"collection1",
:    [junit4]   2> 	  "numShards":"2",
:    [junit4]   2> 	  "core_node_name":null}
:    [junit4]   2> 3192092 T10194 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
:    [junit4]   2> 3192092 T10194 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
:    [junit4]   2> 3192097 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3192099 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
:    [junit4]   2> 3192099 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
:    [junit4]   2> 3192099 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
:    [junit4]   2> 3193088 T10216 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
:    [junit4]   2> 3193089 T10216 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/collection1
:    [junit4]   2> 3193089 T10216 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
:    [junit4]   2> 3193090 T10216 oasc.ZkController.createCollectionZkNode Collection zkNode exists
:    [junit4]   2> 3193090 T10216 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
:    [junit4]   2> 3193093 T10216 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
:    [junit4]   2> 3193093 T10216 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/collection1/'
:    [junit4]   2> 3193094 T10216 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/collection1/lib/.svn/' to classloader
:    [junit4]   2> 3193095 T10216 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/collection1/lib/classes/' to classloader
:    [junit4]   2> 3193095 T10216 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/collection1/lib/README' to classloader
:    [junit4]   2> 3193153 T10216 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
:    [junit4]   2> 3193212 T10216 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
:    [junit4]   2> 3193215 T10216 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
:    [junit4]   2> 3193222 T10216 oass.IndexSchema.readSchema [collection1] Schema name=test
:    [junit4]   2> 3193733 T10216 oass.IndexSchema.readSchema default search field in schema is text
:    [junit4]   2> 3193736 T10216 oass.IndexSchema.readSchema unique key field: id
:    [junit4]   2> 3193738 T10216 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
:    [junit4]   2> 3193743 T10216 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
:    [junit4]   2> 3193788 T10216 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
:    [junit4]   2> 3193788 T10216 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty1-1393996573495/collection1/, dataDir=./org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty1/
:    [junit4]   2> 3193788 T10216 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@13f9690e
:    [junit4]   2> 3193789 T10216 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty1
:    [junit4]   2> 3193789 T10216 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty1/index/
:    [junit4]   2> 3193790 T10216 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty1/index' doesn't exist. Creating new index...
:    [junit4]   2> 3193790 T10216 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty1/index
:    [junit4]   2> 3193791 T10216 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=47, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
:    [junit4]   2> 3193794 T10216 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
:    [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty1/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@55818f9b; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
:    [junit4]   2> 3193794 T10216 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
:    [junit4]   2> 3193800 T10216 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-constraint" (default)
:    [junit4]   2> 3193801 T10216 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-constraint" (default)
:    [junit4]   2> 3193801 T10216 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-float"
:    [junit4]   2> 3193801 T10216 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-float"
:    [junit4]   2> 3193802 T10216 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-failhard"
:    [junit4]   2> 3193802 T10216 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-failhard"
:    [junit4]   2> 3193802 T10216 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
:    [junit4]   2> 3193803 T10216 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
:    [junit4]   2> 3193804 T10216 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
:    [junit4]   2> 3193805 T10216 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
:    [junit4]   2> 3193805 T10216 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
:    [junit4]   2> 3193814 T10216 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
:    [junit4]   2> 3193820 T10216 oasu.CommitTracker.<init> Hard AutoCommit: disabled
:    [junit4]   2> 3193821 T10216 oasu.CommitTracker.<init> Soft AutoCommit: disabled
:    [junit4]   2> 3193822 T10216 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=2, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
:    [junit4]   2> 3193824 T10216 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
:    [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty1/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@55818f9b; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
:    [junit4]   2> 3193824 T10216 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
:    [junit4]   2> 3193824 T10216 oass.SolrIndexSearcher.<init> Opening Searcher@554dfd35[collection1] main
:    [junit4]   2> 3193825 T10217 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@554dfd35[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
:    [junit4]   2> 3193828 T10216 oasc.CoreContainer.registerCore registering core: collection1
:    [junit4]   2> 3193829 T10169 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
:    [junit4]   2> 3193830 T10220 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:57968/iekf collection:collection1 shard:shard2
:    [junit4]   2> 3193830 T10169 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
:    [junit4]   2> 3193832 T10220 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
:    [junit4]   2> 3193842 T10220 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
:    [junit4]   2> 3193846 T10220 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
:    [junit4]   2> 3193846 T10220 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
:    [junit4]   2> ASYNC  NEW_CORE C10078 name=collection1 org.apache.solr.core.SolrCore@b365cbc url=https://127.0.0.1:57968/iekf/collection1 node=127.0.0.1:57968_iekf C10078_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:57968/iekf, core=collection1, node_name=127.0.0.1:57968_iekf}
:    [junit4]   2> 3193846 T10220 C10078 P57968 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:57968/iekf/collection1/
:    [junit4]   2> 3193846 T10220 C10078 P57968 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
:    [junit4]   2> 3193846 T10220 C10078 P57968 oasc.SyncStrategy.syncToMe https://127.0.0.1:57968/iekf/collection1/ has no replicas
:    [junit4]   2> 3193847 T10220 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:57968/iekf/collection1/ shard2
:    [junit4]   2> 3193847 T10220 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
:    [junit4]   2> 3193850 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3193855 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
:    [junit4]   2> 3193861 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3193864 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
:    [junit4]   2> 3193864 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
:    [junit4]   2> 3193868 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
:    [junit4]   2> 3193874 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
:    [junit4]   2> 3193880 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3193986 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
:    [junit4]   2> 3193986 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
:    [junit4]   2> 3193986 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
:    [junit4]   2> 3194022 T10220 oasc.ZkController.register We are https://127.0.0.1:57968/iekf/collection1/ and leader is https://127.0.0.1:57968/iekf/collection1/
:    [junit4]   2> 3194023 T10220 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:57968/iekf
:    [junit4]   2> 3194023 T10220 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
:    [junit4]   2> 3194023 T10220 oasc.ZkController.publish publishing core=collection1 state=active
:    [junit4]   2> 3194023 T10220 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
:    [junit4]   2> 3194025 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3194025 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3194025 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3194026 T10220 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
:    [junit4]   2> 3194029 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
:    [junit4]   2> 3194031 T10194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
:    [junit4]   2> 	  "operation":"state",
:    [junit4]   2> 	  "state":"active",
:    [junit4]   2> 	  "base_url":"https://127.0.0.1:57968/iekf",
:    [junit4]   2> 	  "core":"collection1",
:    [junit4]   2> 	  "roles":null,
:    [junit4]   2> 	  "node_name":"127.0.0.1:57968_iekf",
:    [junit4]   2> 	  "shard":"shard2",
:    [junit4]   2> 	  "collection":"collection1",
:    [junit4]   2> 	  "numShards":"2",
:    [junit4]   2> 	  "core_node_name":"core_node1"}
:    [junit4]   2> 3194036 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3194140 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
:    [junit4]   2> 3194141 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
:    [junit4]   2> 3194140 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
:    [junit4]   2> 3194886 T10169 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
:    [junit4]   2> 3194886 T10169 oejs.Server.doStart jetty-8.1.10.v20130312
:    [junit4]   2> 3194894 T10169 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
:    [junit4]   2> 3194897 T10169 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:57971
:    [junit4]   2> 3194900 T10169 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
:    [junit4]   2> 3194900 T10169 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
:    [junit4]   2> 3194901 T10169 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421
:    [junit4]   2> 3194901 T10169 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/'
:    [junit4]   2> 3194940 T10169 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/solr.xml
:    [junit4]   2> 3195017 T10169 oasc.CoreContainer.<init> New CoreContainer 1957057479
:    [junit4]   2> 3195017 T10169 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/]
:    [junit4]   2> 3195019 T10169 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
:    [junit4]   2> 3195019 T10169 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
:    [junit4]   2> 3195020 T10169 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
:    [junit4]   2> 3195020 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
:    [junit4]   2> 3195021 T10169 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
:    [junit4]   2> 3195021 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
:    [junit4]   2> 3195021 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
:    [junit4]   2> 3195021 T10169 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
:    [junit4]   2> 3195021 T10169 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
:    [junit4]   2> 3195036 T10169 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
:    [junit4]   2> 3195036 T10169 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
:    [junit4]   2> 3195036 T10169 oasc.CoreContainer.load Host Name: 127.0.0.1
:    [junit4]   2> 3195037 T10169 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57960/solr
:    [junit4]   2> 3195037 T10169 oasc.ZkController.checkChrootPath zkHost includes chroot
:    [junit4]   2> 3195038 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
:    [junit4]   2> 3195045 T10231 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45967778 name:ZooKeeperConnection Watcher:127.0.0.1:57960 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
:    [junit4]   2> 3195045 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
:    [junit4]   2> 3195048 T10169 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
:    [junit4]   2> 3195053 T10233 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33135df5 name:ZooKeeperConnection Watcher:127.0.0.1:57960/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
:    [junit4]   2> 3195053 T10169 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
:    [junit4]   2> 3195066 T10169 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
:    [junit4]   2> 3196074 T10169 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57971_iekf
:    [junit4]   2> 3196076 T10169 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57971_iekf
:    [junit4]   2> 3196083 T10193 oascc.ZkStateReader$3.process Updating live nodes... (3)
:    [junit4]   2> 3196083 T10233 oascc.ZkStateReader$3.process Updating live nodes... (3)
:    [junit4]   2> 3196084 T10215 oascc.ZkStateReader$3.process Updating live nodes... (3)
:    [junit4]   2> 3196083 T10202 oascc.ZkStateReader$3.process Updating live nodes... (3)
:    [junit4]   2> 3196101 T10234 oasc.ZkController.publish publishing core=collection1 state=down
:    [junit4]   2> 3196101 T10234 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
:    [junit4]   2> 3196104 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3196104 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3196105 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3196104 T10234 oasc.ZkController.waitForCoreNodeName look for our core node name
:    [junit4]   2> 3196109 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
:    [junit4]   2> 3196110 T10194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
:    [junit4]   2> 	  "operation":"state",
:    [junit4]   2> 	  "state":"down",
:    [junit4]   2> 	  "base_url":"https://127.0.0.1:57971/iekf",
:    [junit4]   2> 	  "core":"collection1",
:    [junit4]   2> 	  "roles":null,
:    [junit4]   2> 	  "node_name":"127.0.0.1:57971_iekf",
:    [junit4]   2> 	  "shard":null,
:    [junit4]   2> 	  "collection":"collection1",
:    [junit4]   2> 	  "numShards":"2",
:    [junit4]   2> 	  "core_node_name":null}
:    [junit4]   2> 3196111 T10194 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
:    [junit4]   2> 3196111 T10194 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
:    [junit4]   2> 3196119 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3196120 T10233 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
:    [junit4]   2> 3196121 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
:    [junit4]   2> 3196121 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
:    [junit4]   2> 3196121 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
:    [junit4]   2> 3197107 T10234 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
:    [junit4]   2> 3197107 T10234 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/collection1
:    [junit4]   2> 3197107 T10234 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
:    [junit4]   2> 3197109 T10234 oasc.ZkController.createCollectionZkNode Collection zkNode exists
:    [junit4]   2> 3197109 T10234 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
:    [junit4]   2> 3197111 T10234 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
:    [junit4]   2> 3197111 T10234 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/collection1/'
:    [junit4]   2> 3197113 T10234 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/collection1/lib/.svn/' to classloader
:    [junit4]   2> 3197113 T10234 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/collection1/lib/classes/' to classloader
:    [junit4]   2> 3197113 T10234 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/collection1/lib/README' to classloader
:    [junit4]   2> 3197173 T10234 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
:    [junit4]   2> 3197236 T10234 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
:    [junit4]   2> 3197239 T10234 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
:    [junit4]   2> 3197245 T10234 oass.IndexSchema.readSchema [collection1] Schema name=test
:    [junit4]   2> 3197730 T10234 oass.IndexSchema.readSchema default search field in schema is text
:    [junit4]   2> 3197734 T10234 oass.IndexSchema.readSchema unique key field: id
:    [junit4]   2> 3197736 T10234 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
:    [junit4]   2> 3197742 T10234 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
:    [junit4]   2> 3197785 T10234 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
:    [junit4]   2> 3197786 T10234 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty2-1393996577421/collection1/, dataDir=./org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty2/
:    [junit4]   2> 3197786 T10234 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@13f9690e
:    [junit4]   2> 3197788 T10234 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty2
:    [junit4]   2> 3197788 T10234 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty2/index/
:    [junit4]   2> 3197789 T10234 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty2/index' doesn't exist. Creating new index...
:    [junit4]   2> 3197790 T10234 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty2/index
:    [junit4]   2> 3197790 T10234 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=47, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
:    [junit4]   2> 3197793 T10234 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
:    [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty2/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@20e77e9a; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
:    [junit4]   2> 3197794 T10234 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
:    [junit4]   2> 3197798 T10234 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-constraint" (default)
:    [junit4]   2> 3197799 T10234 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-constraint" (default)
:    [junit4]   2> 3197799 T10234 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-float"
:    [junit4]   2> 3197800 T10234 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-float"
:    [junit4]   2> 3197800 T10234 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "external-version-failhard"
:    [junit4]   2> 3197800 T10234 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "external-version-failhard"
:    [junit4]   2> 3197800 T10234 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
:    [junit4]   2> 3197801 T10234 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
:    [junit4]   2> 3197801 T10234 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
:    [junit4]   2> 3197802 T10234 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
:    [junit4]   2> 3197804 T10234 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
:    [junit4]   2> 3197814 T10234 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
:    [junit4]   2> 3197821 T10234 oasu.CommitTracker.<init> Hard AutoCommit: disabled
:    [junit4]   2> 3197821 T10234 oasu.CommitTracker.<init> Soft AutoCommit: disabled
:    [junit4]   2> 3197822 T10234 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=2, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
:    [junit4]   2> 3197823 T10234 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
:    [junit4]   2> 		commit{dir=NRTCachingDirectory(NIOFSDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.TestDistribDocBasedVersion-1393996570292/jetty2/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@20e77e9a; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
:    [junit4]   2> 3197823 T10234 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
:    [junit4]   2> 3197823 T10234 oass.SolrIndexSearcher.<init> Opening Searcher@22c9b206[collection1] main
:    [junit4]   2> 3197824 T10235 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@22c9b206[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
:    [junit4]   2> 3197826 T10234 oasc.CoreContainer.registerCore registering core: collection1
:    [junit4]   2> 3197828 T10169 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
:    [junit4]   2> 3197828 T10238 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:57971/iekf collection:collection1 shard:shard1
:    [junit4]   2> 3197828 T10169 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
:    [junit4]   2> 3197830 T10238 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
:    [junit4]   2> 3197838 T10238 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
:    [junit4]   2> 3197841 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3197843 T10238 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
:    [junit4]   2> 3197843 T10238 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
:    [junit4]   2> ASYNC  NEW_CORE C10079 name=collection1 org.apache.solr.core.SolrCore@6c0955af url=https://127.0.0.1:57971/iekf/collection1 node=127.0.0.1:57971_iekf C10079_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:57971/iekf, core=collection1, node_name=127.0.0.1:57971_iekf}
:    [junit4]   2> 3197843 T10238 C10079 P57971 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:57971/iekf/collection1/
:    [junit4]   2> 3197844 T10238 C10079 P57971 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
:    [junit4]   2> 3197844 T10238 C10079 P57971 oasc.SyncStrategy.syncToMe https://127.0.0.1:57971/iekf/collection1/ has no replicas
:    [junit4]   2> 3197844 T10238 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:57971/iekf/collection1/ shard1
:    [junit4]   2> 3197844 T10238 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
:    [junit4]   2> 3197848 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
:    [junit4]   2> 3197855 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3197859 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
:    [junit4]   2> 3197861 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
:    [junit4]   2> 3197861 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
:    [junit4]   2> 3197861 T10233 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
:    [junit4]   2> 3197868 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
:    [junit4]   2> 3197873 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3197978 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
:    [junit4]   2> 3197979 T10233 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
:    [junit4]   2> 3197979 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
:    [junit4]   2> 3197980 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
:    [junit4]   2> 3198021 T10238 oasc.ZkController.register We are https://127.0.0.1:57971/iekf/collection1/ and leader is https://127.0.0.1:57971/iekf/collection1/
:    [junit4]   2> 3198022 T10238 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:57971/iekf
:    [junit4]   2> 3198023 T10238 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
:    [junit4]   2> 3198023 T10238 oasc.ZkController.publish publishing core=collection1 state=active
:    [junit4]   2> 3198023 T10238 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
:    [junit4]   2> 3198025 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3198025 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3198025 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3198027 T10238 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
:    [junit4]   2> 3198029 T10194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
:    [junit4]   2> 3198030 T10194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
:    [junit4]   2> 	  "operation":"state",
:    [junit4]   2> 	  "state":"active",
:    [junit4]   2> 	  "base_url":"https://127.0.0.1:57971/iekf",
:    [junit4]   2> 	  "core":"collection1",
:    [junit4]   2> 	  "roles":null,
:    [junit4]   2> 	  "node_name":"127.0.0.1:57971_iekf",
:    [junit4]   2> 	  "shard":"shard1",
:    [junit4]   2> 	  "collection":"collection1",
:    [junit4]   2> 	  "numShards":"2",
:    [junit4]   2> 	  "core_node_name":"core_node2"}
:    [junit4]   2> 3198038 T10193 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
:    [junit4]   2> 3198149 T10193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
:    [junit4]   2> 3198150 T10233 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
:    [junit4]   2> 3198151 T10215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
:    [junit4]   2> 3198152 T10202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
:    [junit4]   2> 3198958 T10169 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
:    [junit4]   2> 3198959 T10169 oejs.Server.doStart jetty-8.1.10.v20130312
:    [junit4]   2> 3198964 T10169 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
:    [junit4]   2> 3198965 T10169 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:57974
:    [junit4]   2> 3198970 T10169 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
:    [junit4]   2> 3198970 T10169 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
:    [junit4]   2> 3198971 T10169 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty3-1393996581418
:    [junit4]   2> 3198971 T10169 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty3-1393996581418/'
:    [junit4]   2> 3198998 T10169 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty3-1393996581418/solr.xml
:    [junit4]   2> 3199083 T10169 oasc.CoreContainer.<init> New CoreContainer 410602430
:    [junit4]   2> 3199084 T10169 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.TestDistribDocBasedVersion-jetty3-1393996581418/]
:    [junit4]   2> 3199086 T10169 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
:    [junit4]   2> 3199086 T10169 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
:    [junit4]   2> 3199087 T10169 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
:    [junit4]   2> 3199087 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
:    [junit4]   2> 3199087 T10169 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
:    [junit4]   2> 3199087 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
:    [junit4]   2> 3199088 T10169 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
:    [junit4]   2> 3199088 T10169 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
:    [junit4]   2> 3199088 T10169 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
:    [junit4]   2> 3199101 T10169 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
:    [junit4]   2> 3199101 T10169 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
:    [junit4]   2> 3199102 T10169 oasc.CoreContainer.load Host Name: 127.0.0.1
:    [junit4]   2> 3199102 T10169 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57960/solr
:    [junit4]   2> 3199102 T10169 oasc.ZkController.checkChrootPath zkHost includes chroot
:    [junit4]   2> 3199
: 
: [...truncated too long message...]
: 
: ConfigTest, TestQuerySenderListener, AlternateDirectoryTest, TestQuerySenderNoQuery, ResponseLogComponentTest, SolrIndexConfigTest, BadComponentTest, TestMergePolicyConfig, TestBinaryField, MinimalSchemaTest, TestElisionMultitermQuery, TestConfig, OutputWriterTest, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestSolrCoreProperties, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, TestPhraseSuggestions, TestCharFilters, TestCodecSupport, TestXIncludeConfig, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, LegacyHTMLStripCharFilterTest, TestJmxMonitoredMap, TimeZoneUtilsTest, ScriptEngineTest, PluginInfoTest, DateMathParserTest, PreAnalyzedFieldTest, TestSystemIdResolver, TestSuggestSpellingConverter, DateFieldTest, TestSolrJ, TestUtils, TestDocumentBuilder, ZkNodePropsTest, SliceStateTest, SystemInfoHandlerTest, UUIDFieldTest, CircularListTest, CursorPagingTest, NoFacetTest, Abs
 tractAnalyticsFacetTest, FieldFacetExtrasTest, RangeFacetTest, FunctionTest, AliasIntegrationTest, AssignTest, CustomCollectionTest, DeleteInactiveReplicaTest, DistribCursorPagingTest, MigrateRouteKeyTest, RemoteQueryErrorTest, ShardSplitTest, SolrXmlInZkTest, TestDistribDocBasedVersion]
:    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDistribDocBasedVersion -Dtests.seed=1249227945045A2E -Dtests.slow=true -Dtests.locale=ko_KR -Dtests.timezone=America/Monterrey -Dtests.file.encoding=ISO-8859-1
:    [junit4] ERROR   0.00s | TestDistribDocBasedVersion (suite) <<<
:    [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrIndexSearcher opens=13 closes=11
:    [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)
:    [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:418)
:    [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:177)
:    [junit4]    > 	at java.lang.Thread.run(Thread.java:744)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.TestDistribDocBasedVersion: 
:    [junit4]    >    1) Thread[id=10253, name=searcherExecutor-6433-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]
:    [junit4]    >         at sun.misc.Unsafe.park(Native Method)
:    [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
:    [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
:    [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
:    [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
:    [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
:    [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
:    [junit4]    >         at java.lang.Thread.run(Thread.java:744)
:    [junit4]    >    2) Thread[id=10273, name=searcherExecutor-6441-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]
:    [junit4]    >         at sun.misc.Unsafe.park(Native Method)
:    [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
:    [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
:    [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
:    [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
:    [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
:    [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
:    [junit4]    >         at java.lang.Thread.run(Thread.java:744)
:    [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
:    [junit4]    >    1) Thread[id=10273, name=searcherExecutor-6441-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]
:    [junit4]    >         at sun.misc.Unsafe.park(Native Method)
:    [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
:    [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
:    [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
:    [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
:    [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
:    [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
:    [junit4]    >         at java.lang.Thread.run(Thread.java:744)
:    [junit4]    >    2) Thread[id=10253, name=searcherExecutor-6433-thread-1, state=WAITING, group=TGRP-TestDistribDocBasedVersion]
:    [junit4]    >         at sun.misc.Unsafe.park(Native Method)
:    [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
:    [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
:    [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
:    [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
:    [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
:    [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
:    [junit4]    >         at java.lang.Thread.run(Thread.java:744)
:    [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10295, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestDistribDocBasedVersion]
:    [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.AssertionError
:    [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)
:    [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
:    [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
:    [junit4]    > Caused by: java.lang.AssertionError
:    [junit4]    > 	at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:3809)
:    [junit4]    > 	at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:3793)
:    [junit4]    > 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3646)
:    [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
:    [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10296, name=Lucene Merge Thread #0, state=RUNNABLE, group=TGRP-TestDistribDocBasedVersion]
:    [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.AssertionError
:    [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1249227945045A2E]:0)
:    [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:545)
:    [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
:    [junit4]    > Caused by: java.lang.AssertionError
:    [junit4]    > 	at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:3809)
:    [junit4]    > 	at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:3793)
:    [junit4]    > 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3646)
:    [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
:    [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
:    [junit4] Completed in 169.57s, 1 test, 1 failure, 4 errors <<< FAILURES!
: 
: [...truncated 135 lines...]
: BUILD FAILED
: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:465: The following error occurred while executing this line:
: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:445: The following error occurred while executing this line:
: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:490: The following error occurred while executing this line:
: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1274: The following error occurred while executing this line:
: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:907: There were test failures: 377 suites, 1616 tests, 5 suite-level errors, 241 ignored (19 assumptions)
: 
: Total time: 85 minutes 54 seconds
: Build step 'Invoke Ant' marked build as failure
: Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC -XX:-UseSuperWord
: Archiving artifacts
: Recording test results
: Email was triggered for: Failure
: Sending email for trigger: Failure
: 
: 

-Hoss

[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1385 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1385/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:-UseSuperWord

All tests passed

Build Log:
[...truncated 17033 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-map-reduce/test/temp/junit4-J0-20140305_224122_756.syserr
   [junit4] >>> JVM J0: stderr (verbatim) ----
   [junit4] 2014-03-05 22:41:31.201 java[287:6703] Unable to load realm info from SCDynamicStore
   [junit4] <<< JVM J0: EOF ----

[...truncated 13889 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.7
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.7
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.1
[forbidden-apis] Reading API signatures: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/tools/forbiddenApis/base.txt
[forbidden-apis] Reading API signatures: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/tools/forbiddenApis/servlet-api.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning for API signatures and dependencies...
[forbidden-apis] Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[]) [Uses default locale]
[forbidden-apis]   in org.apache.solr.cloud.ZkController (ZkController.java:875)
[forbidden-apis] Scanned 2073 (and 1361 related) class file(s) for forbidden API invocations (in 4.24s), 1 error(s).

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:465: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:64: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:271: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:473: Check for forbidden API calls failed, see log.

Total time: 104 minutes 41 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:-UseSuperWord
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1384 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1384/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseParallelGC -XX:-UseSuperWord

2 tests failed.
FAILED:  org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest.testDistribSearch

Error Message:
IOException occured when talking to server at: https://127.0.0.1:52951/zt_sf/uy/collection1

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrServer$RouteException: IOException occured when talking to server at: https://127.0.0.1:52951/zt_sf/uy/collection1
	at __randomizedtesting.SeedInfo.seed([683888CB73327C8F:E9DE06D3046D1CB3]:0)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.directUpdate(CloudSolrServer.java:351)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:510)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:118)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
	at org.apache.solr.morphlines.solr.SolrServerDocumentLoader.loadBatch(SolrServerDocumentLoader.java:91)
	at org.apache.solr.morphlines.solr.SolrServerDocumentLoader.commitTransaction(SolrServerDocumentLoader.java:79)
	at org.apache.solr.morphlines.solr.LoadSolrBuilder$LoadSolr.doNotify(LoadSolrBuilder.java:95)
	at org.kitesdk.morphline.base.AbstractCommand.notify(AbstractCommand.java:135)
	at org.kitesdk.morphline.base.Connector.notify(Connector.java:57)
	at org.kitesdk.morphline.base.AbstractCommand.doNotify(AbstractCommand.java:150)
	at org.kitesdk.morphline.base.AbstractCommand.notify(AbstractCommand.java:135)
	at org.kitesdk.morphline.base.Connector.notify(Connector.java:57)
	at org.kitesdk.morphline.base.AbstractCommand.doNotify(AbstractCommand.java:150)
	at org.kitesdk.morphline.base.AbstractCommand.notify(AbstractCommand.java:135)
	at org.kitesdk.morphline.base.Connector.notify(Connector.java:57)
	at org.kitesdk.morphline.base.AbstractCommand.doNotify(AbstractCommand.java:150)
	at org.kitesdk.morphline.base.AbstractCommand.notify(AbstractCommand.java:135)
	at org.kitesdk.morphline.base.Connector.notify(Connector.java:57)
	at org.kitesdk.morphline.base.AbstractCommand.doNotify(AbstractCommand.java:150)
	at org.kitesdk.morphline.base.AbstractCommand.notify(AbstractCommand.java:135)
	at org.kitesdk.morphline.base.Connector.notify(Connector.java:57)
	at org.kitesdk.morphline.base.AbstractCommand.doNotify(AbstractCommand.java:150)
	at org.kitesdk.morphline.base.AbstractCommand.notify(AbstractCommand.java:135)
	at org.kitesdk.morphline.base.AbstractCommand.doNotify(AbstractCommand.java:150)
	at org.kitesdk.morphline.base.AbstractCommand.notify(AbstractCommand.java:135)
	at org.kitesdk.morphline.base.Notifications.notify(Notifications.java:96)
	at org.kitesdk.morphline.base.Notifications.notifyCommitTransaction(Notifications.java:61)
	at org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase.commit(AbstractSolrMorphlineZkTestBase.java:103)
	at org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest.doTest(SolrMorphlineZkAvroTest.java:84)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase.testDistribSearch(AbstractSolrMorphlineZkTestBase.java:98)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:52951/zt_sf/uy/collection1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:509)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:302)
	at org.apache.solr.client.solrj.impl.CloudSolrServer$1.call(CloudSolrServer.java:332)
	at org.apache.solr.client.solrj.impl.CloudSolrServer$1.call(CloudSolrServer.java:329)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	... 1 more
Caused by: org.apache.http.NoHttpResponseException: The target server failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:395)
	... 8 more


FAILED:  junit.framework.TestSuite.org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest

Error Message:
ERROR: SolrZkClient opens=15 closes=14

Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
	at __randomizedtesting.SeedInfo.seed([683888CB73327C8F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:433)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:178)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 16947 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-map-reduce/test/temp/junit4-J0-20140305_134109_792.syserr
   [junit4] >>> JVM J0: stderr (verbatim) ----
   [junit4] 2014-03-05 13:41:47.440 java[253:6107] Unable to load realm info from SCDynamicStore
   [junit4] <<< JVM J0: EOF ----

[...truncated 1877 lines...]
   [junit4] Suite: org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest
   [junit4]   2> log4j:WARN No such property [conversionPattern] in org.apache.solr.util.SolrLogLayout.
   [junit4]   2> 2282 T10 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 2366 T10 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /zt_sf/uy
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./solrtest-SolrMorphlineZkAvroTest-1394027041595
   [junit4]   2> 3850 T10 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./solrtest-SolrMorphlineZkAvroTest-1394027041653
   [junit4]   2> 3857 T10 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 3865 T11 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 4067 T10 oasc.ZkTestServer.run start zk server on port:52935
   [junit4]   2> 4358 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4393 T12 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 6218 T17 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@92c3c25 name:ZooKeeperConnection Watcher:127.0.0.1:52935 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 6220 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 6224 T10 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 6294 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 6301 T19 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3669813 name:ZooKeeperConnection Watcher:127.0.0.1:52935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 6301 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 6320 T10 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 6330 T10 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 6342 T10 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 6349 T10 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 6361 T10 oasc.AbstractZkTestCase.putConfig skipping /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase/collection1/conf/solrconfig-tlog.xml because it doesn't exist
   [junit4]   2> 6362 T10 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 6363 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 6377 T10 oasc.AbstractZkTestCase.putConfig skipping /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 6378 T10 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 6379 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 6387 T10 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 6388 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 6409 T10 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 6410 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 6418 T10 oasc.AbstractZkTestCase.putConfig skipping /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 6419 T10 oasc.AbstractZkTestCase.putConfig skipping /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 6419 T10 oasc.AbstractZkTestCase.putConfig skipping /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 6420 T10 oasc.AbstractZkTestCase.putConfig skipping /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 6421 T10 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 6422 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 6450 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 6455 T21 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60cf3fb3 name:ZooKeeperConnection Watcher:127.0.0.1:52935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 6456 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 6463 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 6469 T23 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@164a7140 name:ZooKeeperConnection Watcher:127.0.0.1:52935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 6469 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 6471 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 6491 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/elevate.xml
   [junit4]   2> 6502 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/contractions_ca.txt
   [junit4]   2> 6527 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/contractions_fr.txt
   [junit4]   2> 6538 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/contractions_ga.txt
   [junit4]   2> 6550 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/contractions_it.txt
   [junit4]   2> 6560 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/hyphenations_ga.txt
   [junit4]   2> 6573 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stemdict_nl.txt
   [junit4]   2> 6582 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stoptags_ja.txt
   [junit4]   2> 6593 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_ar.txt
   [junit4]   2> 6604 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_bg.txt
   [junit4]   2> 6617 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_ca.txt
   [junit4]   2> 6626 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_cz.txt
   [junit4]   2> 6636 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_da.txt
   [junit4]   2> 6645 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_de.txt
   [junit4]   2> 6657 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_el.txt
   [junit4]   2> 6667 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_en.txt
   [junit4]   2> 6677 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_es.txt
   [junit4]   2> 6687 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_eu.txt
   [junit4]   2> 6697 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_fa.txt
   [junit4]   2> 6709 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_fi.txt
   [junit4]   2> 6718 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_fr.txt
   [junit4]   2> 6728 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_ga.txt
   [junit4]   2> 6737 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_gl.txt
   [junit4]   2> 6748 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_hi.txt
   [junit4]   2> 6758 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_hu.txt
   [junit4]   2> 6771 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_hy.txt
   [junit4]   2> 6779 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_id.txt
   [junit4]   2> 6791 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_it.txt
   [junit4]   2> 6802 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_ja.txt
   [junit4]   2> 6813 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_lv.txt
   [junit4]   2> 6823 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_nl.txt
   [junit4]   2> 6831 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_no.txt
   [junit4]   2> 6841 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_pt.txt
   [junit4]   2> 6851 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_ro.txt
   [junit4]   2> 6862 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_ru.txt
   [junit4]   2> 6872 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_sv.txt
   [junit4]   2> 6882 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_th.txt
   [junit4]   2> 6891 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/stopwords_tr.txt
   [junit4]   2> 6901 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/lang/userdict_ja.txt
   [junit4]   2> 6911 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 6919 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 6927 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 6935 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 6944 T10 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 8669 T10 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 9434 T10 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 9495 T10 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:52941
   [junit4]   2> 10065 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 10158 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 10158 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756
   [junit4]   2> 10159 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/'
   [junit4]   2> 10710 T10 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/solr.xml
   [junit4]   2> 12106 T10 oasc.CoreContainer.<init> New CoreContainer 749034056
   [junit4]   2> 12107 T10 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/]
   [junit4]   2> 12264 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 12265 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 12276 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 12276 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 12277 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 12278 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 12278 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 12279 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 12279 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 12432 T10 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 12436 T10 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 12444 T10 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 12445 T10 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52935/solr
   [junit4]   2> 12475 T10 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 12477 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 12486 T36 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1dbf9510 name:ZooKeeperConnection Watcher:127.0.0.1:52935 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 12487 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 12511 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 12517 T38 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7cee3a7b name:ZooKeeperConnection Watcher:127.0.0.1:52935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 12518 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 12532 T10 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 12552 T10 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 12590 T10 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 12596 T10 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52941_zt_sf%2Fuy
   [junit4]   2> 12631 T10 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52941_zt_sf%2Fuy
   [junit4]   2> 12647 T10 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 12670 T10 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 12704 T10 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:52941_zt_sf%2Fuy
   [junit4]   2> 12705 T10 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 12713 T10 oasc.Overseer.start Overseer (id=91358956221628421-127.0.0.1:52941_zt_sf%2Fuy-n_0000000000) starting
   [junit4]   2> 12741 T10 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 12821 T40 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 12823 T10 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 12829 T40 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 12839 T10 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 12845 T10 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 12863 T39 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 13214 T41 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 13215 T41 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 13224 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 13224 T41 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 13232 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 13235 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:52941/zt_sf/uy",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52941_zt_sf%2Fuy",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 13236 T39 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 13260 T39 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 13271 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 13275 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 14227 T41 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 14227 T41 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1
   [junit4]   2> 14228 T41 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 14231 T41 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 14231 T41 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 14235 T41 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 14236 T41 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1/'
   [junit4]   2> 14532 T41 oasc.SolrConfig.initLibs Adding specified lib dirs to ClassLoader
   [junit4]   2> 14538 T41 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../contrib/extraction/lib (resolved as: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1/../../../contrib/extraction/lib).
   [junit4]   2> 14539 T41 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1/../../../dist).
   [junit4]   2> 14540 T41 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../contrib/clustering/lib/ (resolved as: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1/../../../contrib/clustering/lib).
   [junit4]   2> 14541 T41 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1/../../../dist).
   [junit4]   2> 14542 T41 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../contrib/langid/lib/ (resolved as: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1/../../../contrib/langid/lib).
   [junit4]   2> 14542 T41 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1/../../../dist).
   [junit4]   2> 14543 T41 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../contrib/velocity/lib (resolved as: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1/../../../contrib/velocity/lib).
   [junit4]   2> 14544 T41 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1/../../../dist).
   [junit4]   2> 14544 T41 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: /non/existent/dir/yields/warning (resolved as: /non/existent/dir/yields/warning).
   [junit4]   2> 14753 T41 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
   [junit4]   2> 15131 T41 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 15153 T41 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 15277 T41 oass.IndexSchema.readSchema [collection1] Schema name=example
   [junit4]   2> 17053 T41 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 17706 T41 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 17730 T41 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 18153 T41 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 18185 T41 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1/, dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1/data/
   [junit4]   2> 18192 T41 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a5bd02f
   [junit4]   2> 18382 T41 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 18382 T41 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher warming in solrconfig.xml}]}
   [junit4]   2> 18491 T41 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1/data
   [junit4]   2> 18492 T41 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1/data/index/
   [junit4]   2> 18493 T41 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 18518 T41 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1/data/index
   [junit4]   2> 18674 T41 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@47fc9ac0 lockFactory=NativeFSLockFactory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 18676 T41 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 18709 T41 oasc.SolrCore.initWriters created json: solr.JSONResponseWriter
   [junit4]   2> 18709 T41 oasc.SolrCore.initWriters adding lazy queryResponseWriter: solr.VelocityResponseWriter
   [junit4]   2> 18714 T41 oasc.SolrCore.initWriters created velocity: solr.VelocityResponseWriter
   [junit4]   2> 18752 T41 oasc.SolrCore.initWriters created xslt: solr.XSLTResponseWriter
   [junit4]   2> 18753 T41 oasr.XSLTResponseWriter.init xsltCacheLifetimeSeconds=5
   [junit4]   2> 19839 T41 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 19933 T41 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 19935 T41 oasc.RequestHandlers.initHandlersFromConfig created /query: solr.SearchHandler
   [junit4]   2> 19953 T41 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 19956 T41 oasc.RequestHandlers.initHandlersFromConfig created /browse: solr.SearchHandler
   [junit4]   2> 19983 T41 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 19999 T41 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.JsonUpdateRequestHandler]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 20001 T41 oasc.RequestHandlers.initHandlersFromConfig created /update/json: solr.JsonUpdateRequestHandler
   [junit4]   2> 20019 T41 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.CSVRequestHandler]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 20020 T41 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: solr.CSVRequestHandler
   [junit4]   2> 20021 T41 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.FieldAnalysisRequestHandler
   [junit4]   2> 20024 T41 oasc.RequestHandlers.initHandlersFromConfig created /analysis/field: solr.FieldAnalysisRequestHandler
   [junit4]   2> 20024 T41 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.DocumentAnalysisRequestHandler
   [junit4]   2> 20024 T41 oasc.RequestHandlers.initHandlersFromConfig created /analysis/document: solr.DocumentAnalysisRequestHandler
   [junit4]   2> 20046 T41 oasc.RequestHandlers.initHandlersFromConfig created /admin/: solr.admin.AdminHandlers
   [junit4]   2> 20064 T41 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 20081 T41 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 20120 T41 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 20120 T41 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.SearchHandler
   [junit4]   2> 20121 T41 oasc.RequestHandlers.initHandlersFromConfig created /spell: solr.SearchHandler
   [junit4]   2> 20121 T41 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.SearchHandler
   [junit4]   2> 20121 T41 oasc.RequestHandlers.initHandlersFromConfig created /tvrh: solr.SearchHandler
   [junit4]   2> 20122 T41 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.SearchHandler
   [junit4]   2> 20122 T41 oasc.RequestHandlers.initHandlersFromConfig created /terms: solr.SearchHandler
   [junit4]   2> 20122 T41 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.SearchHandler
   [junit4]   2> 20123 T41 oasc.RequestHandlers.initHandlersFromConfig created /elevate: solr.SearchHandler
   [junit4]   2> 20245 T41 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 20263 T41 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 20265 T41 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 20351 T41 oasu.CommitTracker.<init> Hard AutoCommit: if uncommited for 60000ms; 
   [junit4]   2> 20351 T41 oasu.CommitTracker.<init> Soft AutoCommit: if uncommited for 1000ms; 
   [junit4]   2> 20358 T41 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@47fc9ac0 lockFactory=NativeFSLockFactory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-controljetty-1394027044756/collection1/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 20359 T41 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 20402 T41 oass.SolrIndexSearcher.<init> Opening Searcher@75283fbd[collection1] main
   [junit4]   2> 20438 T41 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 20531 T41 oass.DirectSolrSpellChecker.init init: {name=default,field=name,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
   [junit4]   2> 20603 T41 oashc.SpellCheckComponent.inform No queryConverter defined, using default converter
   [junit4]   2> 20620 T41 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 20624 T41 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 20823 T41 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 20824 T42 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@75283fbd[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> ASYNC  NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@29b562ae url=https://127.0.0.1:52941/zt_sf/uy/collection1 node=127.0.0.1:52941_zt_sf%2Fuy C0_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:52941/zt_sf/uy, core=collection1, node_name=127.0.0.1:52941_zt_sf%2Fuy}
   [junit4]   2> 21198 T42 C0 P52941 oasc.SolrCore.execute [collection1] webapp=null path=null params={distrib=false&q=static+firstSearcher+warming+in+solrconfig.xml&event=firstSearcher} hits=0 status=0 QTime=364 
   [junit4]   2> 21222 T42 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 21222 T42 oashc.SpellCheckComponent$SpellCheckerListener.newSearcher Loading spell index for spellchecker: default
   [junit4]   2> 21223 T42 oashc.SpellCheckComponent$SpellCheckerListener.newSearcher Loading spell index for spellchecker: wordbreak
   [junit4]   2> 21230 T42 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@75283fbd[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 21238 T41 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 21246 T10 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0
   [junit4]   2> 21247 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 21246 T45 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:52941/zt_sf/uy collection:control_collection shard:shard1
   [junit4]   2> 21268 T45 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 21287 T45 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 21291 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21293 T45 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 21293 T45 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 21297 T45 C0 P52941 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:52941/zt_sf/uy/collection1/
   [junit4]   2> 21299 T45 C0 P52941 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 21300 T45 C0 P52941 oasc.SyncStrategy.syncToMe https://127.0.0.1:52941/zt_sf/uy/collection1/ has no replicas
   [junit4]   2> 21300 T45 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:52941/zt_sf/uy/collection1/ shard1
   [junit4]   2> 21308 T45 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 21318 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 21323 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 21331 T47 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57cd2006 name:ZooKeeperConnection Watcher:127.0.0.1:52935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 21333 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21333 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 21338 T10 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 21339 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 21351 T10 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 21352 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 21361 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21471 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 21477 T47 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 21489 T45 oasc.ZkController.register We are https://127.0.0.1:52941/zt_sf/uy/collection1/ and leader is https://127.0.0.1:52941/zt_sf/uy/collection1/
   [junit4]   2> 21493 T45 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:52941/zt_sf/uy
   [junit4]   2> 21494 T45 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 21495 T45 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 21495 T45 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 21499 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21500 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21501 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21505 T45 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 21509 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 21512 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:52941/zt_sf/uy",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52941_zt_sf%2Fuy",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 21522 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 21629 T47 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 21630 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 22467 T10 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 22468 T10 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 22479 T10 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 22485 T10 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:52945
   [junit4]   2> 22488 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 22488 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 22488 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154
   [junit4]   2> 22489 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/'
   [junit4]   2> 22530 T10 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/solr.xml
   [junit4]   2> 22690 T10 oasc.CoreContainer.<init> New CoreContainer 1211102969
   [junit4]   2> 22690 T10 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/]
   [junit4]   2> 22693 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 22693 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 22694 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 22694 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 22695 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 22695 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 22696 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 22696 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 22697 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 22720 T10 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 22721 T10 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 22721 T10 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 22722 T10 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52935/solr
   [junit4]   2> 22722 T10 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 22726 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 22729 T58 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@205ee81 name:ZooKeeperConnection Watcher:127.0.0.1:52935 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 22737 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 22761 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 22764 T60 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59677282 name:ZooKeeperConnection Watcher:127.0.0.1:52935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 22765 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 22785 T10 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 23795 T10 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52945_zt_sf%2Fuy
   [junit4]   2> 23798 T10 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52945_zt_sf%2Fuy
   [junit4]   2> 23809 T60 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 23811 T47 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 23812 T38 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 23836 T61 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 23836 T61 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 23839 T61 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 23839 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23840 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23840 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23844 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 23846 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:52945/zt_sf/uy",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52945_zt_sf%2Fuy",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 23847 T39 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 23848 T39 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 23854 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23857 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 23858 T47 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 23857 T60 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 24841 T61 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 24842 T61 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1
   [junit4]   2> 24842 T61 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 24848 T61 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 24849 T61 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 24851 T61 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 24851 T61 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1/'
   [junit4]   2> 24930 T61 oasc.SolrConfig.initLibs Adding specified lib dirs to ClassLoader
   [junit4]   2> 24934 T61 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../contrib/extraction/lib (resolved as: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1/../../../contrib/extraction/lib).
   [junit4]   2> 24935 T61 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1/../../../dist).
   [junit4]   2> 24936 T61 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../contrib/clustering/lib/ (resolved as: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1/../../../contrib/clustering/lib).
   [junit4]   2> 24936 T61 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1/../../../dist).
   [junit4]   2> 24937 T61 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../contrib/langid/lib/ (resolved as: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1/../../../contrib/langid/lib).
   [junit4]   2> 24937 T61 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1/../../../dist).
   [junit4]   2> 24938 T61 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../contrib/velocity/lib (resolved as: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1/../../../contrib/velocity/lib).
   [junit4]   2> 24938 T61 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1/../../../dist).
   [junit4]   2> 24939 T61 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: /non/existent/dir/yields/warning (resolved as: /non/existent/dir/yields/warning).
   [junit4]   2> 25213 T61 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
   [junit4]   2> 25321 T61 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 25325 T61 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 25441 T61 oass.IndexSchema.readSchema [collection1] Schema name=example
   [junit4]   2> 26273 T61 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 26360 T61 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 26371 T61 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 26561 T61 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 26563 T61 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1/, dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1/data/
   [junit4]   2> 26567 T61 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a5bd02f
   [junit4]   2> 26573 T61 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 26573 T61 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher warming in solrconfig.xml}]}
   [junit4]   2> 26576 T61 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1/data
   [junit4]   2> 26578 T61 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1/data/index/
   [junit4]   2> 26578 T61 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 26579 T61 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1/data/index
   [junit4]   2> 26586 T61 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@550b7a01 lockFactory=NativeFSLockFactory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 26586 T61 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 26588 T61 oasc.SolrCore.initWriters created json: solr.JSONResponseWriter
   [junit4]   2> 26591 T61 oasc.SolrCore.initWriters adding lazy queryResponseWriter: solr.VelocityResponseWriter
   [junit4]   2> 26593 T61 oasc.SolrCore.initWriters created velocity: solr.VelocityResponseWriter
   [junit4]   2> 26593 T61 oasc.SolrCore.initWriters created xslt: solr.XSLTResponseWriter
   [junit4]   2> 26593 T61 oasr.XSLTResponseWriter.init xsltCacheLifetimeSeconds=5
   [junit4]   2> 26613 T61 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 26616 T61 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
   [junit4]   2> 26622 T61 oasc.RequestHandlers.initHandlersFromConfig created /query: solr.SearchHandler
   [junit4]   2> 26625 T61 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 26627 T61 oasc.RequestHandlers.initHandlersFromConfig created /browse: solr.SearchHandler
   [junit4]   2> 26631 T61 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 26632 T61 oasc.RequestHandlers.initHandlersFromConfig created /update/json: solr.JsonUpdateRequestHandler
   [junit4]   2> 26634 T61 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: solr.CSVRequestHandler
   [junit4]   2> 26635 T61 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.FieldAnalysisRequestHandler
   [junit4]   2> 26635 T61 oasc.RequestHandlers.initHandlersFromConfig created /analysis/field: solr.FieldAnalysisRequestHandler
   [junit4]   2> 26635 T61 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.DocumentAnalysisRequestHandler
   [junit4]   2> 26636 T61 oasc.RequestHandlers.initHandlersFromConfig created /analysis/document: solr.DocumentAnalysisRequestHandler
   [junit4]   2> 26638 T61 oasc.RequestHandlers.initHandlersFromConfig created /admin/: solr.admin.AdminHandlers
   [junit4]   2> 26640 T61 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 26644 T61 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 26650 T61 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 26651 T61 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.SearchHandler
   [junit4]   2> 26652 T61 oasc.RequestHandlers.initHandlersFromConfig created /spell: solr.SearchHandler
   [junit4]   2> 26653 T61 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.SearchHandler
   [junit4]   2> 26653 T61 oasc.RequestHandlers.initHandlersFromConfig created /tvrh: solr.SearchHandler
   [junit4]   2> 26654 T61 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.SearchHandler
   [junit4]   2> 26655 T61 oasc.RequestHandlers.initHandlersFromConfig created /terms: solr.SearchHandler
   [junit4]   2> 26655 T61 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.SearchHandler
   [junit4]   2> 26656 T61 oasc.RequestHandlers.initHandlersFromConfig created /elevate: solr.SearchHandler
   [junit4]   2> 26664 T61 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 26667 T61 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 26674 T61 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 26689 T61 oasu.CommitTracker.<init> Hard AutoCommit: if uncommited for 60000ms; 
   [junit4]   2> 26689 T61 oasu.CommitTracker.<init> Soft AutoCommit: if uncommited for 1000ms; 
   [junit4]   2> 26692 T61 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@550b7a01 lockFactory=NativeFSLockFactory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty1-1394027059154/collection1/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 26693 T61 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 26694 T61 oass.SolrIndexSearcher.<init> Opening Searcher@13289fe4[collection1] main
   [junit4]   2> 26695 T61 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 26695 T61 oass.DirectSolrSpellChecker.init init: {name=default,field=name,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
   [junit4]   2> 26696 T61 oashc.SpellCheckComponent.inform No queryConverter defined, using default converter
   [junit4]   2> 26697 T61 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 26701 T61 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 26723 T61 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 26724 T62 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@13289fe4[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> ASYNC  NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@7b288131 url=https://127.0.0.1:52945/zt_sf/uy/collection1 node=127.0.0.1:52945_zt_sf%2Fuy C1_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:52945/zt_sf/uy, core=collection1, node_name=127.0.0.1:52945_zt_sf%2Fuy}
   [junit4]   2> 26725 T62 C1 P52945 oasc.SolrCore.execute [collection1] webapp=null path=null params={distrib=false&q=static+firstSearcher+warming+in+solrconfig.xml&event=firstSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 26726 T62 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 26726 T62 oashc.SpellCheckComponent$SpellCheckerListener.newSearcher Loading spell index for spellchecker: default
   [junit4]   2> 26726 T62 oashc.SpellCheckComponent$SpellCheckerListener.newSearcher Loading spell index for spellchecker: wordbreak
   [junit4]   2> 26744 T62 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13289fe4[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 26760 T61 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 26776 T10 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0
   [junit4]   2> 26777 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 26787 T65 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:52945/zt_sf/uy collection:collection1 shard:shard3
   [junit4]   2> 26799 T65 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 26841 T65 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 26857 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26863 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 26863 T65 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 26864 T65 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 26866 T65 C1 P52945 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:52945/zt_sf/uy/collection1/
   [junit4]   2> 26866 T65 C1 P52945 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 26867 T65 C1 P52945 oasc.SyncStrategy.syncToMe https://127.0.0.1:52945/zt_sf/uy/collection1/ has no replicas
   [junit4]   2> 26867 T65 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:52945/zt_sf/uy/collection1/ shard3
   [junit4]   2> 26867 T65 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 26883 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26886 T47 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 26886 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 26888 T60 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 26910 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26920 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 26927 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27040 T60 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 27040 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 27040 T47 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 27071 T65 oasc.ZkController.register We are https://127.0.0.1:52945/zt_sf/uy/collection1/ and leader is https://127.0.0.1:52945/zt_sf/uy/collection1/
   [junit4]   2> 27071 T65 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:52945/zt_sf/uy
   [junit4]   2> 27071 T65 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 27072 T65 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 27072 T65 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 27075 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27076 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27076 T65 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 27076 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27085 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 27088 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:52945/zt_sf/uy",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52945_zt_sf%2Fuy",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 27097 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27206 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 27207 T47 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 27208 T60 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 27931 T10 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 27932 T10 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 27940 T10 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 27945 T10 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:52948
   [junit4]   2> 27949 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 27949 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 27950 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty2-1394027064583
   [junit4]   2> 27950 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty2-1394027064583/'
   [junit4]   2> 27995 T10 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty2-1394027064583/solr.xml
   [junit4]   2> 28112 T10 oasc.CoreContainer.<init> New CoreContainer 634703200
   [junit4]   2> 28112 T10 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty2-1394027064583/]
   [junit4]   2> 28115 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 28115 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 28116 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 28116 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 28117 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 28117 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 28117 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 28118 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 28118 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 28138 T10 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 28139 T10 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 28139 T10 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 28139 T10 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52935/solr
   [junit4]   2> 28140 T10 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 28141 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 28146 T76 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@650d154b name:ZooKeeperConnection Watcher:127.0.0.1:52935 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 28147 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 28155 T10 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 28161 T78 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a9c0566 name:ZooKeeperConnection Watcher:127.0.0.1:52935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 28161 T10 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 28175 T10 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 29186 T10 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52948_zt_sf%2Fuy
   [junit4]   2> 29189 T10 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52948_zt_sf%2Fuy
   [junit4]   2> 29196 T38 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 29197 T78 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 29197 T47 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 29197 T60 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 29216 T79 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 29216 T79 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 29219 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29219 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29219 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29219 T79 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 29223 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 29225 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:52948/zt_sf/uy",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52948_zt_sf%2Fuy",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 29226 T39 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 29227 T39 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 29236 T38 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 29239 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 29239 T78 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 29239 T47 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 29239 T60 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 30222 T79 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 30222 T79 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty2-1394027064583/collection1
   [junit4]   2> 30223 T79 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 30224 T79 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 30225 T79 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 30229 T79 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 30229 T79 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-morphlines-core/test/J0/./org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest-jetty2-1394027064

[...truncated too long message...]

-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SolrMorphlineZkAvroTest -Dtests.method=testDistribSearch -Dtests.seed=683888CB73327C8F -Dtests.slow=true -Dtests.locale=fi_FI -Dtests.timezone=Asia/Beirut -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    257s | SolrMorphlineZkAvroTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.CloudSolrServer$RouteException: IOException occured when talking to server at: https://127.0.0.1:52951/zt_sf/uy/collection1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([683888CB73327C8F:E9DE06D3046D1CB3]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.directUpdate(CloudSolrServer.java:351)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:510)
   [junit4]    > 	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:118)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
   [junit4]    > 	at org.apache.solr.morphlines.solr.SolrServerDocumentLoader.loadBatch(SolrServerDocumentLoader.java:91)
   [junit4]    > 	at org.apache.solr.morphlines.solr.SolrServerDocumentLoader.commitTransaction(SolrServerDocumentLoader.java:79)
   [junit4]    > 	at org.apache.solr.morphlines.solr.LoadSolrBuilder$LoadSolr.doNotify(LoadSolrBuilder.java:95)
   [junit4]    > 	at org.kitesdk.morphline.base.AbstractCommand.notify(AbstractCommand.java:135)
   [junit4]    > 	at org.kitesdk.morphline.base.Connector.notify(Connector.java:57)
   [junit4]    > 	at org.kitesdk.morphline.base.AbstractCommand.doNotify(AbstractCommand.java:150)
   [junit4]    > 	at org.kitesdk.morphline.base.AbstractCommand.notify(AbstractCommand.java:135)
   [junit4]    > 	at org.kitesdk.morphline.base.Connector.notify(Connector.java:57)
   [junit4]    > 	at org.kitesdk.morphline.base.AbstractCommand.doNotify(AbstractCommand.java:150)
   [junit4]    > 	at org.kitesdk.morphline.base.AbstractCommand.notify(AbstractCommand.java:135)
   [junit4]    > 	at org.kitesdk.morphline.base.Connector.notify(Connector.java:57)
   [junit4]    > 	at org.kitesdk.morphline.base.AbstractCommand.doNotify(AbstractCommand.java:150)
   [junit4]    > 	at org.kitesdk.morphline.base.AbstractCommand.notify(AbstractCommand.java:135)
   [junit4]    > 	at org.kitesdk.morphline.base.Connector.notify(Connector.java:57)
   [junit4]    > 	at org.kitesdk.morphline.base.AbstractCommand.doNotify(AbstractCommand.java:150)
   [junit4]    > 	at org.kitesdk.morphline.base.AbstractCommand.notify(AbstractCommand.java:135)
   [junit4]    > 	at org.kitesdk.morphline.base.Connector.notify(Connector.java:57)
   [junit4]    > 	at org.kitesdk.morphline.base.AbstractCommand.doNotify(AbstractCommand.java:150)
   [junit4]    > 	at org.kitesdk.morphline.base.AbstractCommand.notify(AbstractCommand.java:135)
   [junit4]    > 	at org.kitesdk.morphline.base.AbstractCommand.doNotify(AbstractCommand.java:150)
   [junit4]    > 	at org.kitesdk.morphline.base.AbstractCommand.notify(AbstractCommand.java:135)
   [junit4]    > 	at org.kitesdk.morphline.base.Notifications.notify(Notifications.java:96)
   [junit4]    > 	at org.kitesdk.morphline.base.Notifications.notifyCommitTransaction(Notifications.java:61)
   [junit4]    > 	at org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase.commit(AbstractSolrMorphlineZkTestBase.java:103)
   [junit4]    > 	at org.apache.solr.morphlines.solr.SolrMorphlineZkAvroTest.doTest(SolrMorphlineZkAvroTest.java:84)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
   [junit4]    > 	at org.apache.solr.morphlines.solr.AbstractSolrMorphlineZkTestBase.testDistribSearch(AbstractSolrMorphlineZkTestBase.java:98)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:52951/zt_sf/uy/collection1
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:509)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:302)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer$1.call(CloudSolrServer.java:332)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer$1.call(CloudSolrServer.java:329)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    > 	... 1 more
   [junit4]    > Caused by: org.apache.http.NoHttpResponseException: The target server failed to respond
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
   [junit4]    > 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
   [junit4]    > 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
   [junit4]    > 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:395)
   [junit4]    > 	... 8 more
   [junit4]   2> 261227 T10 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 261227 T10 oas.SolrTestCaseJ4.endTrackingZkClients ERROR ERROR: SolrZkClient opens=15 closes=14
   [junit4]   2> 261242 T105 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@488b250 name:ZooKeeperConnection Watcher:127.0.0.1:52935/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
   [junit4]   2> 261243 T105 oascc.ConnectionManager.process zkClient has disconnected
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=fi_FI, timezone=Asia/Beirut
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=8,free=113748512,total=245366784
   [junit4]   2> NOTE: All tests run in this JVM: [SolrMorphlineZkAvroTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SolrMorphlineZkAvroTest -Dtests.seed=683888CB73327C8F -Dtests.slow=true -Dtests.locale=fi_FI -Dtests.timezone=Asia/Beirut -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s | SolrMorphlineZkAvroTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=15 closes=14
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([683888CB73327C8F]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:433)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:178)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4] Completed in 261.91s, 1 test, 1 failure, 1 error <<< FAILURES!

[...truncated 19 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:465: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:445: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:209: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:440: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:490: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1274: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:907: There were test failures: 4 suites, 5 tests, 1 suite-level error, 1 error

Total time: 99 minutes 30 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseParallelGC -XX:-UseSuperWord
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure