You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/05/31 11:51:51 UTC

[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 1574 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1574/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 10566 lines...]
    [javac] Compiling 33 source files to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-test-framework/classes/java
    [javac] /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/test-framework/src/java/org/apache/solr/util/RandomMergePolicy.java:36: error: RandomMergePolicy is not abstract and does not override abstract method findForcedDeletesMerges(SegmentInfos,IndexWriter) in MergePolicy
    [javac] public final class RandomMergePolicy extends MergePolicy {
    [javac]              ^
    [javac] /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/test-framework/src/java/org/apache/solr/util/RandomMergePolicy.java:65: error: method findForcedDeletesMerges in class MergePolicy cannot be applied to given types;
    [javac]     return inner.findForcedDeletesMerges(segmentInfos);
    [javac]                 ^
    [javac]   required: SegmentInfos,IndexWriter
    [javac]   found: SegmentInfos
    [javac]   reason: actual and formal argument lists differ in length
    [javac] /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/test-framework/src/java/org/apache/solr/util/RandomMergePolicy.java:72: error: method findForcedMerges in class MergePolicy cannot be applied to given types;
    [javac]     return inner.findForcedMerges(segmentInfos, maxSegmentCount, segmentsToMerge);
    [javac]                 ^
    [javac]   required: SegmentInfos,int,Map<SegmentCommitInfo,Boolean>,IndexWriter
    [javac]   found: SegmentInfos,int,Map<SegmentCommitInfo,Boolean>
    [javac]   reason: actual and formal argument lists differ in length
    [javac] /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/test-framework/src/java/org/apache/solr/util/RandomMergePolicy.java:79: error: method findMerges in class MergePolicy cannot be applied to given types;
    [javac]     return inner.findMerges(mergeTrigger, segmentInfos);
    [javac]                 ^
    [javac]   required: MergeTrigger,SegmentInfos,IndexWriter
    [javac]   found: MergeTrigger,SegmentInfos
    [javac]   reason: actual and formal argument lists differ in length
    [javac] /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/test-framework/src/java/org/apache/solr/util/RandomMergePolicy.java:83: error: cannot find symbol
    [javac]     inner.setIndexWriter(writer);
    [javac]          ^
    [javac]   symbol:   method setIndexWriter(IndexWriter)
    [javac]   location: variable inner of type MergePolicy
    [javac] /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/test-framework/src/java/org/apache/solr/util/RandomMergePolicy.java:90: error: method useCompoundFile in class MergePolicy cannot be applied to given types;
    [javac]     return inner.useCompoundFile(infos, mergedInfo);
    [javac]                 ^
    [javac]   required: SegmentInfos,SegmentCommitInfo,IndexWriter
    [javac]   found: SegmentInfos,SegmentCommitInfo
    [javac]   reason: actual and formal argument lists differ in length
    [javac] Note: Some input files use or override a deprecated API.
    [javac] Note: Recompile with -Xlint:deprecation for details.
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] 6 errors

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:412: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/test-framework/build.xml:41: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1755: Compile failed; see the compiler error output for details.

Total time: 36 minutes 56 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 1577 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1577/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

3 tests failed.
REGRESSION:  org.apache.solr.cloud.ReplicationFactorTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([47860D6A42AAB2F2]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ReplicationFactorTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([47860D6A42AAB2F2]:0)


REGRESSION:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request
	at __randomizedtesting.SeedInfo.seed([47860D6A42AAB2F2:C660837235F5D2CE]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:317)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:650)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1149)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:236)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11221 lines...]
   [junit4] Suite: org.apache.solr.cloud.SyncSliceTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/init-core-data-001
   [junit4]   2> 2865820 T8602 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 2865820 T8602 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 2865827 T8602 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 2865828 T8602 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2865829 T8603 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2865930 T8602 oasc.ZkTestServer.run start zk server on port:52944
   [junit4]   2> 2865933 T8602 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2865949 T8609 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bf3673f name:ZooKeeperConnection Watcher:127.0.0.1:52944 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2865950 T8602 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2865951 T8602 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2865970 T8602 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2865981 T8611 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71bf6b3a name:ZooKeeperConnection Watcher:127.0.0.1:52944/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2865983 T8602 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2865983 T8602 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2865997 T8602 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2866004 T8602 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2866011 T8602 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2866020 T8602 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2866021 T8602 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2866033 T8602 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2866034 T8602 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2866044 T8602 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2866045 T8602 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2866052 T8602 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2866053 T8602 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2866061 T8602 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2866061 T8602 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2866069 T8602 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2866070 T8602 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2866078 T8602 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2866078 T8602 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2866086 T8602 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2866086 T8602 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2866095 T8602 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2866096 T8602 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2866104 T8602 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2866104 T8602 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2866113 T8602 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2866114 T8602 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2867744 T8602 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2867752 T8602 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52947
   [junit4]   2> 2867753 T8602 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2867753 T8602 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2867754 T8602 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-002
   [junit4]   2> 2867754 T8602 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-002/'
   [junit4]   2> 2867803 T8602 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-002/solr.xml
   [junit4]   2> 2867924 T8602 oasc.CoreContainer.<init> New CoreContainer 1505611166
   [junit4]   2> 2867925 T8602 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-002/]
   [junit4]   2> 2867926 T8602 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2867926 T8602 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2867927 T8602 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2867927 T8602 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2867928 T8602 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2867928 T8602 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2867929 T8602 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2867929 T8602 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2867929 T8602 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2867930 T8602 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2867930 T8602 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2867930 T8602 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2867931 T8602 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52944/solr
   [junit4]   2> 2867931 T8602 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2867933 T8602 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2867939 T8622 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c066e19 name:ZooKeeperConnection Watcher:127.0.0.1:52944 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2867940 T8602 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2867946 T8602 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2867949 T8624 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@535cd7f7 name:ZooKeeperConnection Watcher:127.0.0.1:52944/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2867950 T8602 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2867954 T8602 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2867964 T8602 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2867974 T8602 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2867983 T8602 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2867991 T8602 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2868000 T8602 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2868005 T8602 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52947_
   [junit4]   2> 2868007 T8602 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52947_
   [junit4]   2> 2868015 T8602 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2868020 T8602 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2868030 T8602 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:52947_
   [junit4]   2> 2868031 T8602 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2868036 T8602 oasc.Overseer.start Overseer (id=91856342593896451-127.0.0.1:52947_-n_0000000000) starting
   [junit4]   2> 2868045 T8602 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2868073 T8626 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2868074 T8602 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2868080 T8602 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2868086 T8602 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2868101 T8625 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2868447 T8627 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 2868448 T8627 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2868450 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2868450 T8627 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2868455 T8625 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2868456 T8625 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52947",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52947_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2868457 T8625 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 2868457 T8625 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2868464 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2868465 T8624 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2869453 T8627 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2869454 T8627 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 2869457 T8627 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2869458 T8627 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2869463 T8627 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2869464 T8627 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-002/collection1/'
   [junit4]   2> 2869467 T8627 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 2869468 T8627 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 2869470 T8627 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 2869564 T8627 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 2869641 T8627 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2869645 T8627 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2869656 T8627 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2870380 T8627 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2870381 T8627 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2870382 T8627 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2870394 T8627 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2870397 T8627 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2870442 T8627 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2870448 T8627 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2870459 T8627 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2870462 T8627 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2870463 T8627 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2870463 T8627 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2870466 T8627 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2870466 T8627 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2870467 T8627 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 2870467 T8627 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2870467 T8627 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-002/collection1/, dataDir=./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-001/control/data/
   [junit4]   2> 2870468 T8627 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@77bebe00
   [junit4]   2> 2870468 T8627 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-001/control/data
   [junit4]   2> 2870469 T8627 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-001/control/data/index/
   [junit4]   2> 2870469 T8627 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2870469 T8627 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-001/control/data/index
   [junit4]   2> 2870471 T8627 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=50.7294921875, floorSegmentMB=2.1318359375, forceMergeDeletesPctAllowed=26.501220667721196, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2870472 T8627 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3cd240cd lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3242a119),segFN=segments_1,generation=1}
   [junit4]   2> 2870472 T8627 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2870478 T8627 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2870479 T8627 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2870479 T8627 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2870480 T8627 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2870480 T8627 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2870480 T8627 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2870481 T8627 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2870481 T8627 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2870481 T8627 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2870482 T8627 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2870483 T8627 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2870484 T8627 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2870484 T8627 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2870485 T8627 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2870486 T8627 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2870487 T8627 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2870500 T8627 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2870506 T8627 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2870507 T8627 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2870508 T8627 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=25, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4618568158579305]
   [junit4]   2> 2870508 T8627 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3cd240cd lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3242a119),segFN=segments_1,generation=1}
   [junit4]   2> 2870508 T8627 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2870509 T8627 oass.SolrIndexSearcher.<init> Opening Searcher@56531862[collection1] main
   [junit4]   2> 2870509 T8627 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2870512 T8627 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2870513 T8627 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2870514 T8627 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2870514 T8627 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2870514 T8627 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2870516 T8627 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2870516 T8627 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2870516 T8627 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 2870517 T8627 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2870521 T8628 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@56531862[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2870523 T8627 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2870525 T8631 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52947 collection:control_collection shard:shard1
   [junit4]   2> 2870525 T8602 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2870526 T8602 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2870527 T8631 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2870528 T8602 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2870531 T8633 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@447dceab name:ZooKeeperConnection Watcher:127.0.0.1:52944/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2870532 T8602 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2870534 T8602 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2870541 T8602 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2870543 T8631 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2870546 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2870548 T8631 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2870548 T8631 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C5114 name=collection1 org.apache.solr.core.SolrCore@448c56d6 url=http://127.0.0.1:52947/collection1 node=127.0.0.1:52947_ C5114_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:52947, core=collection1, node_name=127.0.0.1:52947_}
   [junit4]   2> 2870548 T8631 C5114 P52947 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52947/collection1/
   [junit4]   2> 2870549 T8631 C5114 P52947 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2870549 T8631 C5114 P52947 oasc.SyncStrategy.syncToMe http://127.0.0.1:52947/collection1/ has no replicas
   [junit4]   2> 2870549 T8631 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52947/collection1/ shard1
   [junit4]   2> 2870549 T8631 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2870554 T8625 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2870563 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2870568 T8624 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2870579 T8625 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2870581 T8633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2870589 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2870693 T8624 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2870696 T8633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2870729 T8631 oasc.ZkController.register We are http://127.0.0.1:52947/collection1/ and leader is http://127.0.0.1:52947/collection1/
   [junit4]   2> 2870729 T8631 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52947
   [junit4]   2> 2870729 T8631 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2870730 T8631 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 2870730 T8631 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2870761 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2870762 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2870762 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2870763 T8631 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2870766 T8625 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2870767 T8625 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52947",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52947_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2870772 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2870878 T8624 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2870878 T8633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2872102 T8602 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2872102 T8602 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2872107 T8602 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52951
   [junit4]   2> 2872108 T8602 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2872108 T8602 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2872108 T8602 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-003
   [junit4]   2> 2872109 T8602 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-003/'
   [junit4]   2> 2872166 T8602 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-003/solr.xml
   [junit4]   2> 2872270 T8602 oasc.CoreContainer.<init> New CoreContainer 750125771
   [junit4]   2> 2872271 T8602 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-003/]
   [junit4]   2> 2872274 T8602 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2872275 T8602 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2872275 T8602 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2872275 T8602 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2872275 T8602 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2872275 T8602 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2872276 T8602 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2872277 T8602 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2872277 T8602 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2872278 T8602 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2872278 T8602 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2872278 T8602 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2872279 T8602 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52944/solr
   [junit4]   2> 2872279 T8602 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2872281 T8602 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2872285 T8644 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6171745b name:ZooKeeperConnection Watcher:127.0.0.1:52944 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2872286 T8602 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2872292 T8602 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2872297 T8646 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15363c0d name:ZooKeeperConnection Watcher:127.0.0.1:52944/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2872298 T8602 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2872321 T8602 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2873333 T8602 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52951_
   [junit4]   2> 2873336 T8602 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52951_
   [junit4]   2> 2873346 T8624 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2873346 T8646 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2873347 T8633 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2873377 T8647 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2873378 T8647 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2873383 T8647 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2873385 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2873385 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2873385 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2873394 T8625 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2873396 T8625 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52951",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52951_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2873396 T8625 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 2873397 T8625 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2873406 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2873409 T8633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2873409 T8624 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2873409 T8646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2874385 T8647 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2874386 T8647 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2874387 T8647 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2874387 T8647 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2874392 T8647 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2874392 T8647 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-003/collection1/'
   [junit4]   2> 2874394 T8647 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 2874395 T8647 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 2874396 T8647 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 2874504 T8647 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 2874599 T8647 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2874602 T8647 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2874617 T8647 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2875566 T8647 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2875567 T8647 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2875568 T8647 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2875591 T8647 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2875596 T8647 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2875635 T8647 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2875643 T8647 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2875653 T8647 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2875657 T8647 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2875658 T8647 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2875658 T8647 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2875662 T8647 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2875662 T8647 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2875663 T8647 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2875663 T8647 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2875664 T8647 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-003/collection1/, dataDir=./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-001/jetty1/
   [junit4]   2> 2875665 T8647 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@77bebe00
   [junit4]   2> 2875667 T8647 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-001/jetty1
   [junit4]   2> 2875667 T8647 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-001/jetty1/index/
   [junit4]   2> 2875668 T8647 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 2875669 T8647 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-001/jetty1/index
   [junit4]   2> 2875669 T8647 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=50.7294921875, floorSegmentMB=2.1318359375, forceMergeDeletesPctAllowed=26.501220667721196, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2875671 T8647 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5d0e692a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4335b3bd),segFN=segments_1,generation=1}
   [junit4]   2> 2875672 T8647 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2875680 T8647 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2875681 T8647 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2875682 T8647 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2875682 T8647 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2875683 T8647 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2875683 T8647 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2875684 T8647 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2875684 T8647 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2875684 T8647 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2875685 T8647 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2875686 T8647 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2875686 T8647 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2875686 T8647 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2875687 T8647 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2875687 T8647 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2875688 T8647 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2875702 T8647 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2875709 T8647 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2875709 T8647 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2875710 T8647 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=25, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4618568158579305]
   [junit4]   2> 2875711 T8647 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5d0e692a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4335b3bd),segFN=segments_1,generation=1}
   [junit4]   2> 2875711 T8647 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2875711 T8647 oass.SolrIndexSearcher.<init> Opening Searcher@7c46071[collection1] main
   [junit4]   2> 2875712 T8647 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2875715 T8647 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2875715 T8647 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2875717 T8647 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2875717 T8647 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2875717 T8647 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2875719 T8647 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2875719 T8647 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2875719 T8647 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 2875720 T8647 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2875724 T8648 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7c46071[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2875727 T8647 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2875729 T8651 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52951 collection:collection1 shard:shard1
   [junit4]   2> 2875730 T8602 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2875730 T8602 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2875731 T8651 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2875749 T8651 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2875754 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2875757 T8651 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2875757 T8651 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C5115 name=collection1 org.apache.solr.core.SolrCore@24edd95a url=http://127.0.0.1:52951/collection1 node=127.0.0.1:52951_ C5115_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:52951, core=collection1, node_name=127.0.0.1:52951_}
   [junit4]   2> 2875758 T8651 C5115 P52951 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52951/collection1/
   [junit4]   2> 2875758 T8651 C5115 P52951 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2875759 T8651 C5115 P52951 oasc.SyncStrategy.syncToMe http://127.0.0.1:52951/collection1/ has no replicas
   [junit4]   2> 2875759 T8651 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52951/collection1/ shard1
   [junit4]   2> 2875759 T8651 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2875765 T8625 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2875780 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2875788 T8646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2875789 T8624 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2875793 T8633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2875807 T8625 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2875813 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2875919 T8624 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2875920 T8633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2875920 T8646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2875956 T8651 oasc.ZkController.register We are http://127.0.0.1:52951/collection1/ and leader is http://127.0.0.1:52951/collection1/
   [junit4]   2> 2875956 T8651 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52951
   [junit4]   2> 2875957 T8651 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2875958 T8651 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2875958 T8651 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2875962 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2875962 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2875963 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2875964 T8651 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2875969 T8625 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2875970 T8625 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52951",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52951_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2875977 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2876084 T8646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2876084 T8624 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2876085 T8633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2877379 T8602 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2877380 T8602 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2877384 T8602 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52954
   [junit4]   2> 2877384 T8602 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2877385 T8602 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2877385 T8602 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-004
   [junit4]   2> 2877386 T8602 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-004/'
   [junit4]   2> 2877444 T8602 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-004/solr.xml
   [junit4]   2> 2877545 T8602 oasc.CoreContainer.<init> New CoreContainer 1007817807
   [junit4]   2> 2877546 T8602 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-004/]
   [junit4]   2> 2877550 T8602 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2877550 T8602 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2877551 T8602 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2877552 T8602 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2877552 T8602 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2877552 T8602 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2877553 T8602 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2877553 T8602 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2877553 T8602 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2877554 T8602 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2877554 T8602 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2877555 T8602 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2877555 T8602 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52944/solr
   [junit4]   2> 2877556 T8602 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2877557 T8602 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2877563 T8662 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46f36d58 name:ZooKeeperConnection Watcher:127.0.0.1:52944 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2877564 T8602 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2877574 T8602 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2877581 T8664 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44117d3 name:ZooKeeperConnection Watcher:127.0.0.1:52944/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2877582 T8602 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2877611 T8602 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2878623 T8602 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52954_
   [junit4]   2> 2878627 T8602 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52954_
   [junit4]   2> 2878638 T8664 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2878638 T8633 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2878639 T8624 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2878639 T8646 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2878660 T8665 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2878661 T8665 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2878664 T8665 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2878664 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2878665 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2878665 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2878669 T8625 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2878671 T8625 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52954",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52954_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2878671 T8625 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
   [junit4]   2> 2878671 T8625 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2878679 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2878681 T8664 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2878681 T8633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2878681 T8624 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2878681 T8646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2879667 T8665 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2879668 T8665 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2879669 T8665 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2879670 T8665 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2879673 T8665 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2879674 T8665 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-004/collection1/'
   [junit4]   2> 2879676 T8665 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 2879677 T8665 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 2879678 T8665 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 2879767 T8665 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 2879882 T8665 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2879886 T8665 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2879896 T8665 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2880749 T8665 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2880750 T8665 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 2880751 T8665 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2880765 T8665 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2880770 T8665 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2880808 T8665 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2880816 T8665 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2880823 T8665 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2880828 T8665 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2880829 T8665 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2880829 T8665 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2880833 T8665 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2880833 T8665 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2880834 T8665 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2880834 T8665 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2880834 T8665 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-004/collection1/, dataDir=./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-001/jetty2/
   [junit4]   2> 2880835 T8665 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@77bebe00
   [junit4]   2> 2880835 T8665 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-001/jetty2
   [junit4]   2> 2880836 T8665 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-001/jetty2/index/
   [junit4]   2> 2880836 T8665 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 2880836 T8665 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-001/jetty2/index
   [junit4]   2> 2880837 T8665 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=30, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=50.7294921875, floorSegmentMB=2.1318359375, forceMergeDeletesPctAllowed=26.501220667721196, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2880838 T8665 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1206c083 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c55216c),segFN=segments_1,generation=1}
   [junit4]   2> 2880838 T8665 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2880846 T8665 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2880846 T8665 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2880847 T8665 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2880847 T8665 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2880847 T8665 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2880848 T8665 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2880848 T8665 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2880849 T8665 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2880849 T8665 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2880850 T8665 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2880850 T8665 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2880851 T8665 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2880851 T8665 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2880852 T8665 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2880852 T8665 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2880854 T8665 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2880865 T8665 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2880871 T8665 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2880872 T8665 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2880872 T8665 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=25, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4618568158579305]
   [junit4]   2> 2880873 T8665 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1206c083 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c55216c),segFN=segments_1,generation=1}
   [junit4]   2> 2880874 T8665 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2880874 T8665 oass.SolrIndexSearcher.<init> Opening Searcher@65ab7a2c[collection1] main
   [junit4]   2> 2880875 T8665 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2880879 T8665 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2880879 T8665 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2880881 T8665 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2880882 T8665 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2880882 T8665 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2880884 T8665 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2880884 T8665 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2880884 T8665 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 2880884 T8665 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2880889 T8666 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@65ab7a2c[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2880892 T8665 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2880894 T8669 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52954 collection:collection1 shard:shard1
   [junit4]   2> 2880895 T8602 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2880896 T8602 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2880902 T8669 oasc.ZkController.register We are http://127.0.0.1:52954/collection1/ and leader is http://127.0.0.1:52951/collection1/
   [junit4]   2> 2880902 T8669 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52954
   [junit4]   2> 2880903 T8669 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 2880903 T8669 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C5116 name=collection1 org.apache.solr.core.SolrCore@48f604d5 url=http://127.0.0.1:52954/collection1 node=127.0.0.1:52954_ C5116_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:52954, core=collection1, node_name=127.0.0.1:52954_}
   [junit4]   2> 2880904 T8670 C5116 P52954 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 2880905 T8670 C5116 P52954 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 2880905 T8670 C5116 P52954 oasc.RecoveryStrategy.doRecovery Publishing state of core collection1 as recovering, leader is http://127.0.0.1:52951/collection1/ and I am http://127.0.0.1:52954/collection1/
   [junit4]   2> 2880905 T8670 C5116 P52954 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
   [junit4]   2> 2880906 T8670 C5116 P52954 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2880909 T8669 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2880912 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2880913 T8670 C5116 P52954 oasc.RecoveryStrategy.sendPrepRecoveryCmd Sending prep recovery command to http://127.0.0.1:52951; WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1%3A52954_&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
   [junit4]   2> 2880922 T8639 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 2880927 T8639 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2880928 T8625 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2880931 T8639 oasha.CoreAdminHandler.handleWaitForStateAction Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 2880932 T8639 oasha.CoreAdminHandler.handleWaitForStateAction In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:52954_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"state":"down","base_url":"http://127.0.0.1:52954","core":"collection1","node_name":"127.0.0.1:52954_"}
   [junit4]   2> 2880932 T8625 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52954",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52954_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 2880940 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2880946 T8633 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2880946 T8624 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2880947 T8646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2880949 T8664 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2881933 T8639 oasha.CoreAdminHandler.handleWaitForStateAction In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:52954_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"state":"recovering","base_url":"http://127.0.0.1:52954","core":"collection1","node_name":"127.0.0.1:52954_"}
   [junit4]   2> 2881934 T8639 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
   [junit4]   2> 2881934 T8639 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node2&nodeName=127.0.0.1:52954_&wt=javabin&state=recovering&core=collection1&checkLive=true&action=PREPRECOVERY&version=2&onlyIfLeader=true&onlyIfLeaderActive=true} status=0 QTime=1012 
   [junit4]   2> 2882602 T8602 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 2882602 T8602 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2882606 T8602 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52958
   [junit4]   2> 2882607 T8602 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2882607 T8602 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2882608 T8602 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-005
   [junit4]   2> 2882609 T8602 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-005/'
   [junit4]   2> 2882664 T8602 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-005/solr.xml
   [junit4]   2> 2882771 T8602 oasc.CoreContainer.<init> New CoreContainer 1274055719
   [junit4]   2> 2882771 T8602 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.SyncSliceTest-47860D6A42AAB2F2-001/tempDir-005/]
   [junit4]   2> 2882774 T8602 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2882775 T8602 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2882778 T8602 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2882778 T8602 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2882779 T8602 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2882779 T8602 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2882780 T8602 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2882781 T8602 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2882781 T8602 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2882782 T8602 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2882783 T8602 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2882784 T8602 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2882784 T8602 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52944/solr
   [junit4]   2> 2882785 T8602 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2882787 T8602 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2882795 T8682 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d817b64 name:ZooKeeperConnection Watcher:127.0.0.1:52944 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2882795 T8602 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2882800 T8602 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2882803 T8684 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@604abc13 name:ZooKeeperConnection Watcher:127.0.0.1:52944/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2882803 T8602 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2882824 T8602 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2883833 T8602 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52958_
   [junit4]   2> 2883835 T8602 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52958_
   [junit4]   2> 2883845 T8646 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2883845 T8633 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2883846 T8664 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2883846 T8684 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2883846 T8624 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2883865 T8685 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2883865 T8685 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2883867 T8624 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2883867 T8685 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2883872 T8625 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2883874 T8625 o

[...truncated too long message...]

ava:360)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 11445870 T12422 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 11445878 T12422 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> NOTE: test params are: codec=Lucene3x, sim=DefaultSimilarity, locale=bg_BG, timezone=Canada/Saskatchewan
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_55 (64-bit)/cpus=2,threads=31,free=313938768,total=519438336
   [junit4]   2> NOTE: All tests run in this JVM: [TestRecovery, InfoHandlerTest, PreAnalyzedFieldTest, TestDocSet, TestQuerySenderListener, LukeRequestHandlerTest, RankQueryTest, TestStressReorder, BasicDistributedZkTest, QueryParsingTest, TestDistributedMissingSort, DOMUtilTest, CopyFieldTest, CoreContainerCoreInitFailuresTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestSuggestSpellingConverter, DateFieldTest, SOLR749Test, TestDefaultSearchFieldResource, TestSerializedLuceneMatchVersion, OverseerStatusTest, XmlUpdateRequestHandlerTest, FastVectorHighlighterTest, TestCustomSort, DocValuesMissingTest, LeaderElectionTest, PluginInfoTest, TestDynamicFieldCollectionResource, TestSolrXmlPersistence, OutputWriterTest, HdfsSyncSliceTest, TestDynamicFieldResource, TestArbitraryIndexDir, TestRealTimeGet, TestCollationField, ZkCLITest, RequestHandlersTest, TestUpdate, ZkNodePropsTest, TestIndexSearcher, TestStressLucene, TestSolrQueryParserResource, TestHighlightDedupGrouping, TestDocumentBuilder, TestCloudInspectUtil, TestPHPSerializedResponseWriter, EchoParamsTest, SpellCheckCollatorTest, TestUniqueKeyFieldResource, TestReplicationHandler, TestNRTOpen, SolrCoreTest, TestLRUCache, JsonLoaderTest, TestReloadAndDeleteDocs, TestOverriddenPrefixQueryForCustomFieldType, TestFuzzyAnalyzedSuggestions, ScriptEngineTest, AddBlockUpdateTest, CoreMergeIndexesAdminHandlerTest, LeaderElectionIntegrationTest, TestIBSimilarityFactory, DistributedExpandComponentTest, TestJmxMonitoredMap, AtomicUpdatesTest, TestHighFrequencyDictionaryFactory, ZkSolrClientTest, DirectSolrSpellCheckerTest, TestClassNameShortening, SliceStateUpdateTest, StatsComponentTest, SpellingQueryConverterTest, DirectUpdateHandlerOptimizeTest, TestAnalyzedSuggestions, OpenExchangeRatesOrgProviderTest, DistributedSpellCheckComponentTest, WordBreakSolrSpellCheckerTest, ShowFileRequestHandlerTest, TestPartialUpdateDeduplication, MergeStrategyTest, TestManagedStopFilterFactory, SolrIndexConfigTest, DocValuesTest, AsyncMigrateRouteKeyTest, BasicZkTest, DocumentBuilderTest, CurrencyFieldOpenExchangeTest, DistributedQueryComponentCustomSortTest, TestRequestStatusCollectionAPI, TestRandomFaceting, TestRandomMergePolicy, DeleteInactiveReplicaTest, SolrCoreCheckLockOnStartupTest, TestSystemIdResolver, BadIndexSchemaTest, ResourceLoaderTest, TestPseudoReturnFields, TestBM25SimilarityFactory, PathHierarchyTokenizerFactoryTest, MultiThreadedOCPTest, QueryEqualityTest, TestElisionMultitermQuery, TestAddFieldRealTimeGet, TestCSVResponseWriter, JSONWriterTest, BadComponentTest, CursorMarkTest, TestComponentsName, TestSweetSpotSimilarityFactory, SimpleFacetsTest, PeerSyncTest, TestConfigSets, DocExpirationUpdateProcessorFactoryTest, TestRangeQuery, PreAnalyzedUpdateProcessorTest, TestCopyFieldCollectionResource, StandardRequestHandlerTest, SampleTest, TestBinaryField, IndexBasedSpellCheckerTest, SpellPossibilityIteratorTest, SolrCmdDistributorTest, TestStandardQParsers, TimeZoneUtilsTest, ParsingFieldUpdateProcessorsTest, TestQueryTypes, OverseerRolesTest, TestBinaryResponseWriter, TestExtendedDismaxParser, TestShortCircuitedRequests, TestSolrDeletionPolicy2, TestSolrIndexConfig, OpenCloseCoreStressTest, TestManagedResourceStorage, TestFieldTypeResource, TestBadConfig, TestInfoStreamLogging, BasicDistributedZk2Test, FileBasedSpellCheckerTest, TestShardHandlerFactory, MBeansHandlerTest, CoreAdminHandlerTest, RecoveryZkTest, DocValuesMultiTest, CollectionsAPIAsyncDistributedZkTest, CurrencyFieldXmlFileTest, TestLazyCores, TestCloudManagedSchema, TestZkChroot, HighlighterTest, CursorPagingTest, CacheHeaderTest, TestFunctionQuery, TestLFUCache, DistributedQueryElevationComponentTest, TestDFRSimilarityFactory, SolrPluginUtilsTest, TermsComponentTest, LoggingHandlerTest, SimplePostToolTest, CustomCollectionTest, TestManagedResource, TestSchemaVersionResource, ChaosMonkeyNothingIsSafeTest, TestManagedSchema, SolrInfoMBeanTest, TestMiniSolrCloudCluster, DistributedTermsComponentTest, ShardSplitTest, TestMergePolicyConfig, SchemaVersionSpecificBehaviorTest, CSVRequestHandlerTest, TestSearcherReuse, SystemInfoHandlerTest, DeleteShardTest, TestFaceting, StatelessScriptUpdateProcessorFactoryTest, UpdateParamsTest, EnumFieldTest, SearchHandlerTest, SoftAutoCommitTest, TestConfig, AnalysisErrorHandlingTest, TestNoOpRegenerator, CoreAdminRequestStatusTest, ConvertedLegacyTest, TestComplexPhraseQParserPlugin, FieldMutatingUpdateProcessorTest, TriLevelCompositeIdRoutingTest, HdfsRecoveryZkTest, TestLuceneMatchVersion, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, SyncSliceTest, OverseerTest, ShardRoutingTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, TestRandomDVFaceting, ShardRoutingCustomTest, TestDistributedSearch, AutoCommitTest, TestDistributedGrouping, TestHashPartitioner, TermVectorComponentDistributedTest, ZkControllerTest, TestJoin, TestMultiCoreConfBootstrap, TestReload, TestStressVersions, HardAutoCommitTest, TestCoreContainer, TestSolr4Spatial, SpellCheckComponentTest, TestGroupingSearch, QueryElevationComponentTest, TestSort, TestFiltering, BasicFunctionalityTest, DirectUpdateHandlerTest, OverseerCollectionProcessorTest, SolrIndexSplitterTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, SuggesterFSTTest, SolrRequestParserTest, TestFoldingMultitermQuery, SuggesterTSTTest, SuggesterTest, TestTrie, SpatialFilterTest, SuggesterWFSTTest, TestCSVLoader, PolyFieldTest, NoCacheHeaderTest, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, DefaultValueUpdateProcessorTest, SortByFunctionTest, TestRemoteStreaming, DistanceFunctionTest, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, DebugComponentTest, TestSurroundQueryParser, DisMaxRequestHandlerTest, TestQueryUtils, TestWriterPerf, TestReversedWildcardFilterFactory, DocumentAnalysisRequestHandlerTest, PrimitiveFieldTypeTest, TestOmitPositions, TestValueSourceCache, TermVectorComponentTest, TestIndexingPerformance, MoreLikeThisHandlerTest, RequiredFieldsTest, FieldAnalysisRequestHandlerTest, TestSolrQueryParser, IndexSchemaRuntimeFieldTest, RegexBoostProcessorTest, IndexSchemaTest, TestJmxIntegration, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, UniqFieldsUpdateProcessorFactoryTest, BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, HighlighterConfigTest, AlternateDirectoryTest, TestQuerySenderNoQuery, ResponseLogComponentTest, TestStressRecovery, MultiTermTest, TestSearchPerf, NumericFieldsTest, MinimalSchemaTest, ExternalFileFieldSortTest, TestSolrCoreProperties, TestPostingsSolrHighlighter, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestPhraseSuggestions, TestCharFilters, TestCodecSupport, SynonymTokenizerTest, TestXIncludeConfig, TestPerFieldSimilarity, TestNumberUtils, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestDefaultSimilarityFactory, TestFastWriter, LegacyHTMLStripCharFilterTest, TestFastOutputStream, URLClassifyProcessorTest, TestFastLRUCache, DateMathParserTest, TestSolrXMLSerializer, PrimUtilsTest, RAMDirectoryFactoryTest, ClusterStateTest, TestSolrJ, TestUtils, SliceStateTest, UUIDFieldTest, FileUtilsTest, CircularListTest, TestRTGBase, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, TestCollationKeyRangeQueries, AliasIntegrationTest, AssignTest, ConnectionManagerTest, DeleteReplicaTest, DistribCursorPagingTest, DistribDocExpirationUpdateProcessorTest, HttpPartitionTest, MigrateRouteKeyTest, RemoteQueryErrorTest, ReplicationFactorTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ReplicationFactorTest -Dtests.seed=47860D6A42AAB2F2 -Dtests.slow=true -Dtests.locale=bg_BG -Dtests.timezone=Canada/Saskatchewan -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s | ReplicationFactorTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([47860D6A42AAB2F2]:0)
   [junit4] Completed in 7224.06s, 1 test, 2 errors <<< FAILURES!

[...truncated 197 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1297: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:921: There were test failures: 396 suites, 1640 tests, 1 suite-level error, 2 errors, 201 ignored (14 assumptions)

Total time: 232 minutes 26 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.8.0) - Build # 1576 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1576/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([B08FF94C05DFBC3D:316977547280DC01]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testSolrJAPICalls(CollectionsAPIDistributedZkTest.java:265)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:203)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.CustomCollectionTest.testDistribSearch

Error Message:
IOException occured when talking to server at: https://127.0.0.1:56470/ksz/routeFieldColl

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:56470/ksz/routeFieldColl
	at __randomizedtesting.SeedInfo.seed([B08FF94C05DFBC3D:316977547280DC01]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:567)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
	at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:168)
	at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:146)
	at org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForHashRouter(CustomCollectionTest.java:414)
	at org.apache.solr.cloud.CustomCollectionTest.doTest(CustomCollectionTest.java:128)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.http.NoHttpResponseException: The target server failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:452)
	... 48 more




Build Log:
[...truncated 10980 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/init-core-data-001
   [junit4]   2> 1009408 T3592 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1009408 T3592 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1009413 T3592 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1009414 T3592 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1009415 T3593 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1009516 T3592 oasc.ZkTestServer.run start zk server on port:54824
   [junit4]   2> 1009517 T3592 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1009974 T3599 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b03fe50 name:ZooKeeperConnection Watcher:127.0.0.1:54824 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1009975 T3592 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1009976 T3592 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1009986 T3592 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1009989 T3601 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d0c6c6e name:ZooKeeperConnection Watcher:127.0.0.1:54824/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1009990 T3592 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1009990 T3592 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1009998 T3592 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1010004 T3592 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1010010 T3592 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1010017 T3592 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1010018 T3592 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1010027 T3592 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1010028 T3592 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1010035 T3592 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1010035 T3592 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1010042 T3592 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1010043 T3592 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1010049 T3592 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1010050 T3592 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1010056 T3592 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1010056 T3592 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1010063 T3592 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1010063 T3592 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1010070 T3592 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1010070 T3592 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1010076 T3592 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1010077 T3592 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1010083 T3592 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1010084 T3592 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1010089 T3592 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1010090 T3592 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 1011651 T3592 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1011688 T3592 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54827
   [junit4]   2> 1011689 T3592 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1011689 T3592 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1011689 T3592 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-002
   [junit4]   2> 1011690 T3592 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-002/'
   [junit4]   2> 1011739 T3592 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-002/solr.xml
   [junit4]   2> 1011758 T3592 oasc.CoreContainer.<init> New CoreContainer 1043960833
   [junit4]   2> 1011759 T3592 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-002/]
   [junit4]   2> 1011760 T3592 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1011760 T3592 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1011761 T3592 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1011761 T3592 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1011762 T3592 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1011762 T3592 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1011762 T3592 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1011763 T3592 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1011763 T3592 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1011764 T3592 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1011764 T3592 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1011765 T3592 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1011765 T3592 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54824/solr
   [junit4]   2> 1011766 T3592 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1011768 T3592 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1011771 T3612 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1468d5bc name:ZooKeeperConnection Watcher:127.0.0.1:54824 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1011772 T3592 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1011778 T3592 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1011782 T3614 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a9925c7 name:ZooKeeperConnection Watcher:127.0.0.1:54824/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1011782 T3592 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1011786 T3592 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1011795 T3592 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1011803 T3592 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1011811 T3592 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1011819 T3592 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1011826 T3592 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1011830 T3592 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54827_
   [junit4]   2> 1011832 T3592 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54827_
   [junit4]   2> 1011840 T3592 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1011845 T3592 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1011855 T3592 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:54827_
   [junit4]   2> 1011856 T3592 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1011862 T3592 oasc.Overseer.start Overseer (id=91854486849716227-127.0.0.1:54827_-n_0000000000) starting
   [junit4]   2> 1011870 T3592 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1011896 T3616 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1011897 T3592 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1011903 T3592 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1011907 T3592 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1011916 T3615 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1011920 T3592 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1011920 T3592 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1011923 T3592 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1011926 T3619 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75636d13 name:ZooKeeperConnection Watcher:127.0.0.1:54824/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1011926 T3592 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1011929 T3592 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1011936 T3592 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1013508 T3592 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1013509 T3592 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1013513 T3592 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54831
   [junit4]   2> 1013514 T3592 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1013515 T3592 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1013515 T3592 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-003
   [junit4]   2> 1013515 T3592 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-003/'
   [junit4]   2> 1013556 T3592 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-003/solr.xml
   [junit4]   2> 1013575 T3592 oasc.CoreContainer.<init> New CoreContainer 1318021065
   [junit4]   2> 1013576 T3592 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-003/]
   [junit4]   2> 1013577 T3592 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1013577 T3592 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1013578 T3592 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1013578 T3592 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1013578 T3592 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1013579 T3592 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1013579 T3592 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1013580 T3592 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1013580 T3592 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1013581 T3592 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1013581 T3592 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1013581 T3592 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1013583 T3592 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54824/solr
   [junit4]   2> 1013583 T3592 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1013585 T3592 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1013589 T3630 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33ae8f70 name:ZooKeeperConnection Watcher:127.0.0.1:54824 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1013590 T3592 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1013596 T3592 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1013598 T3632 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@daee87b name:ZooKeeperConnection Watcher:127.0.0.1:54824/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1013599 T3592 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1013617 T3592 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1014626 T3592 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54831_
   [junit4]   2> 1014629 T3592 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54831_
   [junit4]   2> 1014636 T3614 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1014639 T3632 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1014639 T3619 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1014646 T3592 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1014646 T3592 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1016213 T3592 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1016215 T3592 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1016220 T3592 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54834
   [junit4]   2> 1016222 T3592 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1016222 T3592 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1016222 T3592 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-004
   [junit4]   2> 1016223 T3592 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-004/'
   [junit4]   2> 1016265 T3592 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-004/solr.xml
   [junit4]   2> 1016286 T3592 oasc.CoreContainer.<init> New CoreContainer 707013230
   [junit4]   2> 1016286 T3592 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-004/]
   [junit4]   2> 1016287 T3592 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1016287 T3592 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1016288 T3592 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1016288 T3592 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1016288 T3592 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1016289 T3592 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1016290 T3592 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1016290 T3592 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1016290 T3592 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1016291 T3592 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1016292 T3592 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1016292 T3592 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1016293 T3592 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54824/solr
   [junit4]   2> 1016293 T3592 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1016295 T3592 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1016299 T3644 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31964b90 name:ZooKeeperConnection Watcher:127.0.0.1:54824 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1016300 T3592 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1016306 T3592 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1016309 T3646 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e1ed4b0 name:ZooKeeperConnection Watcher:127.0.0.1:54824/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1016309 T3592 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1016328 T3592 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1017336 T3592 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54834_
   [junit4]   2> 1017338 T3592 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54834_
   [junit4]   2> 1017346 T3619 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1017346 T3646 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1017346 T3632 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1017347 T3614 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1017358 T3592 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1017359 T3592 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1018879 T3592 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1018880 T3592 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1018885 T3592 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54837
   [junit4]   2> 1018886 T3592 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1018887 T3592 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1018887 T3592 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-005
   [junit4]   2> 1018888 T3592 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-005/'
   [junit4]   2> 1018926 T3592 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-005/solr.xml
   [junit4]   2> 1018946 T3592 oasc.CoreContainer.<init> New CoreContainer 2031317685
   [junit4]   2> 1018947 T3592 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-005/]
   [junit4]   2> 1018948 T3592 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1018948 T3592 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1018948 T3592 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1018949 T3592 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1018950 T3592 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1018950 T3592 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1018950 T3592 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1018951 T3592 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1018951 T3592 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1018952 T3592 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1018952 T3592 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1018953 T3592 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1018953 T3592 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54824/solr
   [junit4]   2> 1018954 T3592 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1018956 T3592 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1018959 T3658 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ea94641 name:ZooKeeperConnection Watcher:127.0.0.1:54824 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1018960 T3592 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1018966 T3592 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1018969 T3660 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b38fc1c name:ZooKeeperConnection Watcher:127.0.0.1:54824/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1018969 T3592 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1018987 T3592 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1019996 T3592 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54837_
   [junit4]   2> 1019999 T3592 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54837_
   [junit4]   2> 1020006 T3632 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1020007 T3646 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1020007 T3619 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1020008 T3660 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1020010 T3614 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1020024 T3592 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1020024 T3592 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1021668 T3592 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 1021669 T3592 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1021673 T3592 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54840
   [junit4]   2> 1021673 T3592 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1021674 T3592 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1021674 T3592 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-006
   [junit4]   2> 1021675 T3592 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-006/'
   [junit4]   2> 1021720 T3592 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-006/solr.xml
   [junit4]   2> 1021740 T3592 oasc.CoreContainer.<init> New CoreContainer 922802694
   [junit4]   2> 1021740 T3592 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-006/]
   [junit4]   2> 1021741 T3592 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1021741 T3592 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1021742 T3592 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1021742 T3592 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1021742 T3592 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1021743 T3592 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1021744 T3592 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1021744 T3592 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1021744 T3592 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1021745 T3592 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1021745 T3592 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1021746 T3592 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1021747 T3592 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54824/solr
   [junit4]   2> 1021748 T3592 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1021752 T3592 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1021755 T3672 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@40f30a49 name:ZooKeeperConnection Watcher:127.0.0.1:54824 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1021757 T3592 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1021765 T3592 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1021768 T3674 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@386d89de name:ZooKeeperConnection Watcher:127.0.0.1:54824/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1021768 T3592 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1021787 T3592 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1022795 T3592 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54840_
   [junit4]   2> 1022799 T3592 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54840_
   [junit4]   2> 1022811 T3646 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1022811 T3614 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1022812 T3619 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1022812 T3674 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1022813 T3660 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1022813 T3632 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 1022824 T3592 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1022824 T3592 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1022853 T3628 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&collection.configName=conf1&router.field=myOwnField&name=solrj_collection&router.name=compositeId&action=CREATE&numShards=2&wt=javabin&version=2
   [junit4]   2> 1022856 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1022856 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1022857 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1022857 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1022857 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1022857 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1022858 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1022858 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1022858 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1022859 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1022859 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1022860 T3616 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 1022863 T3676 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField",
   [junit4]   2> 	  "router.name":"compositeId"}
   [junit4]   2> 1022865 T3676 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 1022865 T3676 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 1022879 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1022884 T3615 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1022886 T3615 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 1022886 T3615 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 1022893 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1022895 T3646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1022896 T3632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1022896 T3619 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1022896 T3660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1022897 T3674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1022896 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1022980 T3676 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection solrj_collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 1022980 T3676 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:54834_
   [junit4]   2> 1022983 T3676 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:54827_
   [junit4]   2> 1022987 T3676 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:54831_
   [junit4]   2> 1022990 T3641 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=solrj_collection_shard1_replica1&action=CREATE&numShards=2&collection=solrj_collection&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1022995 T3608 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=solrj_collection_shard1_replica2&action=CREATE&numShards=2&collection=solrj_collection&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1022995 T3676 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:54840_
   [junit4]   2> 1023000 T3641 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 1023001 T3623 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=solrj_collection_shard2_replica1&action=CREATE&numShards=2&collection=solrj_collection&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1023004 T3608 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 1023005 T3668 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=solrj_collection_shard2_replica2&action=CREATE&numShards=2&collection=solrj_collection&shard=shard2&wt=javabin&version=2
   [junit4]   2> 1023006 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023006 T3641 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1023009 T3608 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1023010 T3623 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 1023014 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023014 T3668 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 1023014 T3623 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1023014 T3615 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1023017 T3668 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1023017 T3615 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54834",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54834_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1023026 T3615 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54827",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54827_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1023031 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023033 T3615 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54831",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54831_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1023039 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023042 T3615 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54840",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54840_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1023048 T3614 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1023154 T3614 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1023155 T3632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1023156 T3660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1023156 T3674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1023155 T3619 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1023155 T3646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1024010 T3641 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 1024016 T3608 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 1024017 T3623 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 1024018 T3623 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-003/solr.xml
   [junit4]   2> 1024019 T3623 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1024017 T3641 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-004/solr.xml
   [junit4]   2> 1024019 T3641 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1024018 T3668 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 1024018 T3608 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-002/solr.xml
   [junit4]   2> 1024021 T3668 oasc.SolrXMLCoresLocator.doPersist Persisted core descriptions to /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-006/solr.xml
   [junit4]   2> 1024021 T3668 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1024020 T3623 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1024024 T3623 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1024021 T3641 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1024029 T3641 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1024021 T3608 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1024029 T3623 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1024028 T3668 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1024033 T3668 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1024031 T3623 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-003/solrj_collection_shard2_replica1/'
   [junit4]   2> 1024035 T3608 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1024036 T3608 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1024040 T3608 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1024041 T3608 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-002/solrj_collection_shard1_replica2/'
   [junit4]   2> 1024045 T3641 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1024045 T3641 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-004/solrj_collection_shard1_replica1/'
   [junit4]   2> 1024046 T3668 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1024046 T3668 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-006/solrj_collection_shard2_replica2/'
   [junit4]   2> 1024127 T3608 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 1024145 T3668 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 1024132 T3623 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 1024179 T3623 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1024183 T3668 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1024184 T3623 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1024191 T3623 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 1024195 T3668 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1024195 T3641 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 1024228 T3641 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1024234 T3608 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1024235 T3641 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1024249 T3668 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 1024259 T3608 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1024266 T3608 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 1024307 T3641 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 1024622 T3623 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1024623 T3623 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1024625 T3623 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1024639 T3623 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1024640 T3623 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1024719 T3608 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1024721 T3608 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1024735 T3608 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1024759 T3641 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1024759 T3641 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1024761 T3641 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1024764 T3608 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1024767 T3608 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1024767 T3623 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1024776 T3641 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1024777 T3641 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1024788 T3623 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1024793 T3623 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1024797 T3623 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1024798 T3623 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1024798 T3623 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1024803 T3623 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1024803 T3623 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1024805 T3623 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using configuration from collection solrj_collection
   [junit4]   2> 1024805 T3623 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1024806 T3623 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-003/solrj_collection_shard2_replica1/, dataDir=./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-003/solrj_collection_shard2_replica1/data/
   [junit4]   2> 1024808 T3623 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@410dd6bb
   [junit4]   2> 1024809 T3623 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-003/solrj_collection_shard2_replica1/data
   [junit4]   2> 1024811 T3668 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1024812 T3668 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1024813 T3668 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1024825 T3608 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1024826 T3623 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-003/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 1024828 T3623 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory './solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-003/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1024828 T3668 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1024829 T3623 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-003/solrj_collection_shard2_replica1/data/index
   [junit4]   2> 1024830 T3623 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=36, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=20.2119140625, floorSegmentMB=1.8046875, forceMergeDeletesPctAllowed=7.0417185467779255, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6139926052389201
   [junit4]   2> 1024830 T3668 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1024835 T3641 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1024838 T3623 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4162bc15 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@64dd512c),segFN=segments_1,generation=1}
   [junit4]   2> 1024839 T3623 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1024840 T3641 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1024845 T3641 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1024847 T3608 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1024852 T3641 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1024854 T3641 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1024854 T3641 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1024857 T3608 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1024858 T3641 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1024858 T3641 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1024860 T3641 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using configuration from collection solrj_collection
   [junit4]   2> 1024861 T3641 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1024859 T3623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1024861 T3641 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-004/solrj_collection_shard1_replica1/, dataDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-004/solrj_collection_shard1_replica1/data/
   [junit4]   2> 1024863 T3623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1024865 T3623 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1024863 T3641 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@410dd6bb
   [junit4]   2> 1024866 T3623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1024867 T3641 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-004/solrj_collection_shard1_replica1/data
   [junit4]   2> 1024868 T3623 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1024869 T3623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1024870 T3641 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-004/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 1024871 T3641 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-004/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1024867 T3608 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1024873 T3641 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-004/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 1024873 T3608 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1024874 T3608 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1024871 T3623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1024875 T3623 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1024876 T3623 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1024874 T3641 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=36, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=20.2119140625, floorSegmentMB=1.8046875, forceMergeDeletesPctAllowed=7.0417185467779255, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6139926052389201
   [junit4]   2> 1024880 T3623 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1024881 T3608 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1024882 T3608 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1024882 T3608 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using configuration from collection solrj_collection
   [junit4]   2> 1024883 T3608 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1024882 T3623 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1024884 T3623 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1024885 T3623 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1024886 T3623 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1024887 T3623 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1024887 T3623 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1024893 T3668 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1024884 T3608 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-002/solrj_collection_shard1_replica2/, dataDir=./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-002/solrj_collection_shard1_replica2/data/
   [junit4]   2> 1024894 T3608 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@410dd6bb
   [junit4]   2> 1024894 T3641 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@510979f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@17d24c16),segFN=segments_1,generation=1}
   [junit4]   2> 1024895 T3641 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1024897 T3623 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1024895 T3608 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-002/solrj_collection_shard1_replica2/data
   [junit4]   2> 1024899 T3641 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1024900 T3641 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1024900 T3641 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1024901 T3641 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1024901 T3641 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1024902 T3641 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1024903 T3641 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1024904 T3641 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1024904 T3641 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1024901 T3668 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1024909 T3623 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1024909 T3623 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1024911 T3608 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-002/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 1024911 T3608 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory './solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-002/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1024912 T3623 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=30.88671875, floorSegmentMB=1.6044921875, forceMergeDeletesPctAllowed=6.758888695186749, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1400653153555357
   [junit4]   2> 1024914 T3641 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1024915 T3641 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1024915 T3641 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1024916 T3641 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1024912 T3608 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-002/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 1024920 T3608 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=36, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=20.2119140625, floorSegmentMB=1.8046875, forceMergeDeletesPctAllowed=7.0417185467779255, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6139926052389201
   [junit4]   2> 1024920 T3668 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1024922 T3623 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4162bc15 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@64dd512c),segFN=segments_1,generation=1}
   [junit4]   2> 1024923 T3623 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1024924 T3623 oass.SolrIndexSearcher.<init> Opening Searcher@7922bad[solrj_collection_shard2_replica1] main
   [junit4]   2> 1024929 T3623 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1024926 T3641 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1024930 T3608 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@38ae7331 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5f704986),segFN=segments_1,generation=1}
   [junit4]   2> 1024931 T3608 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1024932 T3623 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1024933 T3623 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1024934 T3623 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1024934 T3623 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1024931 T3641 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1024935 T3668 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1024936 T3668 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1024935 T3623 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1024935 T3608 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1024937 T3608 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1024938 T3608 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1024939 T3608 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1024940 T3608 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1024940 T3608 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1024941 T3608 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1024942 T3608 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1024942 T3608 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1024943 T3608 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1024944 T3608 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1024944 T3608 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1024945 T3608 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1024947 T3608 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1024936 T3668 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1024949 T3668 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1024949 T3668 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1024950 T3668 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using configuration from collection solrj_collection
   [junit4]   2> 1024952 T3668 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1024952 T3668 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-006/solrj_collection_shard2_replica2/, dataDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-006/solrj_collection_shard2_replica2/data/
   [junit4]   2> 1024953 T3668 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@410dd6bb
   [junit4]   2> 1024954 T3668 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-006/solrj_collection_shard2_replica2/data
   [junit4]   2> 1024954 T3608 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1024955 T3608 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1024957 T3608 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1024964 T3608 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1024965 T3608 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1024966 T3668 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-006/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 1024966 T3668 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-006/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1024967 T3668 oasc.CachingDirectoryFactory.get return new directory for ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.CollectionsAPIDistributedZkTest-B08FF94C05DFBC3D-001/tempDir-006/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 1024968 T3668 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=36, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=20.2119140625, floorSegmentMB=1.8046875, forceMergeDeletesPctAllowed=7.0417185467779255, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6139926052389201
   [junit4]   2> 1024968 T3608 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=30.88671875, floorSegmentMB=1.6044921875, forceMergeDeletesPctAllowed=6.758888695186749, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1400653153555357
   [junit4]   2> 1024973 T3608 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockD

[...truncated too long message...]

E: test params are: codec=Lucene46: {shard_s=PostingsFormat(name=Memory doPackFST= true), other_tl1=PostingsFormat(name=Memory doPackFST= true), multiDefault=MockFixedIntBlock(blockSize=1651), intDefault=PostingsFormat(name=Memory doPackFST= true), range_facet_sl=PostingsFormat(name=SimpleText), range_facet_si=MockFixedIntBlock(blockSize=1651), range_facet_l=PostingsFormat(name=Memory doPackFST= true), _version_=PostingsFormat(name=Memory doPackFST= true), a_t=PostingsFormat(name=Memory doPackFST= true), a_si=MockFixedIntBlock(blockSize=1651), id=Pulsing41(freqCutoff=5 minBlockSize=11 maxBlockSize=95), text=PostingsFormat(name=SimpleText), _route_=PostingsFormat(name=SimpleText), timestamp=PostingsFormat(name=Memory doPackFST= true)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=es_CO, timezone=UTC
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=2,threads=1,free=72178136,total=319111168
   [junit4]   2> NOTE: All tests run in this JVM: [SimpleFacetsTest, TestOverriddenPrefixQueryForCustomFieldType, TestRecoveryHdfs, UpdateParamsTest, SynonymTokenizerTest, DocExpirationUpdateProcessorFactoryTest, TestStressVersions, TestJmxIntegration, TestAnalyzeInfixSuggestions, BasicFunctionalityTest, DocumentBuilderTest, ChaosMonkeySafeLeaderTest, TestDocumentBuilder, TestCSVLoader, XmlUpdateRequestHandlerTest, TestFieldResource, BinaryUpdateRequestHandlerTest, AddBlockUpdateTest, TestSolrQueryParserResource, TestSchemaResource, TestStressReorder, TestSolrDeletionPolicy2, TestPHPSerializedResponseWriter, UUIDFieldTest, TestBinaryField, TestFuzzyAnalyzedSuggestions, TestInitQParser, SolrIndexSplitterTest, PreAnalyzedUpdateProcessorTest, FileBasedSpellCheckerTest, SchemaVersionSpecificBehaviorTest, TestNumberUtils, TestSolrQueryParser, TestSearcherReuse, OverseerRolesTest, MigrateRouteKeyTest, ShardRoutingTest, ClusterStateUpdateTest, HdfsSyncSliceTest, DefaultValueUpdateProcessorTest, TestPerFieldSimilarity, TestDefaultSearchFieldResource, DistributedTermsComponentTest, SolrCoreCheckLockOnStartupTest, JSONWriterTest, ChaosMonkeyNothingIsSafeTest, TestRealTimeGet, TestShardHandlerFactory, TestDistributedGrouping, CachingDirectoryFactoryTest, TestFieldTypeCollectionResource, TestAtomicUpdateErrorCases, CoreContainerCoreInitFailuresTest, SolrXmlInZkTest, TestComplexPhraseQParserPlugin, TestFunctionQuery, AtomicUpdatesTest, TestManagedSchemaFieldResource, JsonLoaderTest, RequiredFieldsTest, TestSolrXmlPersistor, TestReversedWildcardFilterFactory, DisMaxRequestHandlerTest, TestCopyFieldCollectionResource, PolyFieldTest, EchoParamsTest, TestSweetSpotSimilarityFactory, TestSolrJ, UnloadDistributedZkTest, CSVRequestHandlerTest, OpenExchangeRatesOrgProviderTest, DocValuesMultiTest, TestSolrXMLSerializer, TestHighFrequencyDictionaryFactory, ZkControllerTest, TestHashPartitioner, TestSurroundQueryParser, ShardSplitTest, TestFieldTypeResource, DistributedExpandComponentTest, ClusterStateTest, TestSolrXml, TestSchemaSimilarityResource, TestReloadAndDeleteDocs, CurrencyFieldXmlFileTest, TestExpandComponent, TestMiniSolrCloudCluster, IndexBasedSpellCheckerTest, OverseerCollectionProcessorTest, TestLuceneMatchVersion, SuggestComponentTest, PrimitiveFieldTypeTest, TestSearchPerf, ResponseHeaderTest, ConvertedLegacyTest, PeerSyncTest, TestManagedSchema, TestCoreDiscovery, BasicZkTest, TestDistributedMissingSort, TestInfoStreamLogging, NotRequiredUniqueKeyTest, HdfsWriteToMultipleCollectionsTest, ExternalFileFieldSortTest, StatsComponentTest, CollectionsAPIDistributedZkTest, HdfsBasicDistributedZk2Test, TestCollectionAPI, TestSerializedLuceneMatchVersion, TestRecovery, TestFoldingMultitermQuery, TestImplicitCoreProperties, TestSimpleQParserPlugin, MoreLikeThisHandlerTest, SuggesterTest, PathHierarchyTokenizerFactoryTest, DeleteInactiveReplicaTest, TestLMJelinekMercerSimilarityFactory, NumericFieldsTest, RegexBoostProcessorTest, InfoHandlerTest, OverseerTest, DirectSolrConnectionTest, BJQParserTest, TestCharFilters, DocValuesMissingTest, DistributedQueryComponentOptimizationTest, TestArbitraryIndexDir, BlockDirectoryTest, TestManagedResourceStorage, TestRestManager, ChangedSchemaMergeTest, CacheHeaderTest, TestSolrQueryParserDefaultOperatorResource, CopyFieldTest, SOLR749Test, SystemInfoHandlerTest, SliceStateUpdateTest, TestExtendedDismaxParser, SolrCoreTest, TestSolrIndexConfig, ZkNodePropsTest, OverseerStatusTest, HighlighterConfigTest, XsltUpdateRequestHandlerTest, TestPartialUpdateDeduplication, IndexSchemaRuntimeFieldTest, TestQueryTypes, TestMergePolicyConfig, DistribCursorPagingTest, TestNRTOpen, ZkCLITest, TestCloudManagedSchema, TestStandardQParsers, SoftAutoCommitTest, URLClassifyProcessorTest, TestDynamicFieldCollectionResource, FullSolrCloudDistribCmdsTest, TestTrie, TestRandomFaceting, TestCollationFieldDocValues, TestManagedSynonymFilterFactory, TestDocBasedVersionConstraints, MBeansHandlerTest, NoCacheHeaderTest, HdfsCollectionsAPIDistributedZkTest, UpdateRequestProcessorFactoryTest, TestRTGBase, TestDocSet, QueryElevationComponentTest, SearchHandlerTest, AnalysisErrorHandlingTest, BlockCacheTest, TestFaceting, TestStressLucene, DirectUpdateHandlerTest, TestSolr4Spatial, TestStressRecovery, QueryParsingTest, StressHdfsTest, TestNonDefinedSimilarityFactory, TestFieldCollectionResource, TestFileDictionaryLookup, MinimalSchemaTest, AutoCommitTest, UniqFieldsUpdateProcessorFactoryTest, SolrCmdDistributorTest, BasicDistributedZkTest, BasicDistributedZk2Test, SyncSliceTest, OpenCloseCoreStressTest, LeaderElectionIntegrationTest, RecoveryZkTest, TestReplicationHandler, LeaderElectionTest, ZkSolrClientTest, TestZkChroot, TestRandomDVFaceting, ShardRoutingCustomTest, TestDistributedSearch, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, TestJoin, TestMultiCoreConfBootstrap, TestReload, HardAutoCommitTest, TestRangeQuery, TestCoreContainer, SpellCheckComponentTest, TestGroupingSearch, BadIndexSchemaTest, TestSort, TestFiltering, TestLazyCores, TestBadConfig, TestIndexSearcher, HighlighterTest, ShowFileRequestHandlerTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, SimplePostToolTest, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, SpellCheckCollatorTest, SuggesterFSTTest, CoreAdminHandlerTest, SolrRequestParserTest, DocValuesTest, SuggesterTSTTest, SpatialFilterTest, SuggesterWFSTTest, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, TestUpdate, FieldMutatingUpdateProcessorTest, TestWordDelimiterFilterFactory, QueryEqualityTest, DirectUpdateHandlerOptimizeTest, StatelessScriptUpdateProcessorFactoryTest, SortByFunctionTest, TestRemoteStreaming, DistanceFunctionTest, TestSolrDeletionPolicy1, SolrInfoMBeanTest, DebugComponentTest, LukeRequestHandlerTest, TestQueryUtils, StandardRequestHandlerTest, TestWriterPerf, DirectSolrSpellCheckerTest, DocumentAnalysisRequestHandlerTest, TestOmitPositions, TermsComponentTest, RequestHandlersTest, TestValueSourceCache, TermVectorComponentTest, TestIndexingPerformance, FieldAnalysisRequestHandlerTest, FastVectorHighlighterTest, LoggingHandlerTest, SolrPluginUtilsTest, IndexSchemaTest, TestCollationField, ReturnFieldsTest, TestCSVResponseWriter, TestAnalyzedSuggestions, PingRequestHandlerTest, TestComponentsName, TestBinaryResponseWriter, TestLFUCache, TestQuerySenderListener, AlternateDirectoryTest, TestQuerySenderNoQuery, ResponseLogComponentTest, SolrIndexConfigTest, BadComponentTest, MultiTermTest, SampleTest, TestElisionMultitermQuery, TestConfig, OutputWriterTest, TestSolrCoreProperties, TestPostingsSolrHighlighter, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCodecSupport, TestXIncludeConfig, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestIBSimilarityFactory, TestBM25SimilarityFactory, TestDefaultSimilarityFactory, TestFastWriter, LegacyHTMLStripCharFilterTest, TestJmxMonitoredMap, TimeZoneUtilsTest, ResourceLoaderTest, TestFastOutputStream, ScriptEngineTest, PluginInfoTest, TestFastLRUCache, DateMathParserTest, PreAnalyzedFieldTest, TestSystemIdResolver, PrimUtilsTest, TestSuggestSpellingConverter, DateFieldTest, SpellingQueryConverterTest, DOMUtilTest, RAMDirectoryFactoryTest, TestLRUCache, TestUtils, SliceStateTest, FileUtilsTest, CircularListTest, CursorPagingTest, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, TestHighlightDedupGrouping, TestCollationKeyRangeQueries, AliasIntegrationTest, AssignTest, AsyncMigrateRouteKeyTest, CollectionsAPIAsyncDistributedZkTest, ConnectionManagerTest, CustomCollectionTest]
   [junit4] Completed in 250.36s, 1 test, 1 error <<< FAILURES!

[...truncated 185 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1297: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:921: There were test failures: 396 suites, 1640 tests, 1 error, 1 failure, 41 ignored (14 assumptions)

Total time: 104 minutes 52 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 1575 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1575/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:55851/collection1, http://127.0.0.1:55857/collection1, http://127.0.0.1:55887/collection1, http://127.0.0.1:55881/collection1, http://127.0.0.1:55847/collection1, http://127.0.0.1:55838/collection1, http://127.0.0.1:55841/collection1, http://127.0.0.1:55844/collection1, http://127.0.0.1:55869/collection1, http://127.0.0.1:55893/collection1, http://127.0.0.1:55875/collection1, http://127.0.0.1:55863/collection1]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:55851/collection1, http://127.0.0.1:55857/collection1, http://127.0.0.1:55887/collection1, http://127.0.0.1:55881/collection1, http://127.0.0.1:55847/collection1, http://127.0.0.1:55838/collection1, http://127.0.0.1:55841/collection1, http://127.0.0.1:55844/collection1, http://127.0.0.1:55869/collection1, http://127.0.0.1:55893/collection1, http://127.0.0.1:55875/collection1, http://127.0.0.1:55863/collection1]
	at __randomizedtesting.SeedInfo.seed([6FEB1EEAFFA9C377:EE0D90F288F6A34B]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:319)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:650)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1149)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:55881/collection1, http://127.0.0.1:55847/collection1, http://127.0.0.1:55841/collection1, http://127.0.0.1:55838/collection1, http://127.0.0.1:55869/collection1, http://127.0.0.1:55863/collection1]
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:554)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:340)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:301)
	... 47 more




Build Log:
[...truncated 11221 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/init-core-data-001
   [junit4]   2> 2126148 T6933 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 2126148 T6933 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 2126176 T6933 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 2126176 T6933 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2126177 T6934 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2126278 T6933 oasc.ZkTestServer.run start zk server on port:55831
   [junit4]   2> 2126283 T6933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2126293 T6940 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71ce4ae1 name:ZooKeeperConnection Watcher:127.0.0.1:55831 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2126294 T6933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2126295 T6933 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2126321 T6933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2126328 T6942 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@184d4074 name:ZooKeeperConnection Watcher:127.0.0.1:55831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2126328 T6933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2126329 T6933 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2126340 T6933 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2126357 T6933 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2126361 T6933 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2126369 T6933 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2126370 T6933 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2126381 T6933 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 2126381 T6933 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2126388 T6933 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2126388 T6933 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2126395 T6933 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2126396 T6933 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2126404 T6933 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2126404 T6933 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2126413 T6933 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2126413 T6933 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2126422 T6933 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2126424 T6933 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2126433 T6933 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2126433 T6933 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2126445 T6933 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2126446 T6933 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2126456 T6933 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2126457 T6933 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2126465 T6933 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2126465 T6933 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2128024 T6933 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2128030 T6933 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55834
   [junit4]   2> 2128031 T6933 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2128031 T6933 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2128032 T6933 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-002
   [junit4]   2> 2128032 T6933 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-002/'
   [junit4]   2> 2128071 T6933 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-002/solr.xml
   [junit4]   2> 2128159 T6933 oasc.CoreContainer.<init> New CoreContainer 504666239
   [junit4]   2> 2128160 T6933 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-002/]
   [junit4]   2> 2128162 T6933 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2128162 T6933 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2128163 T6933 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2128163 T6933 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2128163 T6933 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2128163 T6933 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2128164 T6933 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2128164 T6933 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2128164 T6933 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2128165 T6933 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2128165 T6933 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2128166 T6933 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2128166 T6933 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55831/solr
   [junit4]   2> 2128167 T6933 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2128168 T6933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2128172 T6953 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a27c946 name:ZooKeeperConnection Watcher:127.0.0.1:55831 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2128173 T6933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2128181 T6933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2128184 T6955 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@265f36e7 name:ZooKeeperConnection Watcher:127.0.0.1:55831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2128184 T6933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2128188 T6933 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2128197 T6933 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2128205 T6933 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2128215 T6933 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2128224 T6933 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2128232 T6933 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2128236 T6933 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55834_
   [junit4]   2> 2128238 T6933 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55834_
   [junit4]   2> 2128245 T6933 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2128252 T6933 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2128261 T6933 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55834_
   [junit4]   2> 2128261 T6933 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2128267 T6933 oasc.Overseer.start Overseer (id=91852180029702147-127.0.0.1:55834_-n_0000000000) starting
   [junit4]   2> 2128274 T6933 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2128302 T6957 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2128303 T6933 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2128310 T6933 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2128315 T6933 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2128324 T6956 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2128335 T6958 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 2128336 T6958 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2128338 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2128338 T6958 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2128343 T6956 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2128345 T6956 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55834",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55834_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2128345 T6956 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 2128345 T6956 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2128354 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2128356 T6955 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2129340 T6958 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2129340 T6958 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 2129342 T6958 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2129343 T6958 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2129346 T6958 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2129347 T6958 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-002/collection1/'
   [junit4]   2> 2129349 T6958 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 2129350 T6958 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 2129352 T6958 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 2129458 T6958 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 2129536 T6958 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2129541 T6958 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2129559 T6958 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2130288 T6958 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2130293 T6958 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2130297 T6958 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2130307 T6958 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2130379 T6958 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 2130380 T6958 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 2130380 T6958 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/control/data/
   [junit4]   2> 2130380 T6958 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@cc77302
   [junit4]   2> 2130382 T6958 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/control/data
   [junit4]   2> 2130383 T6958 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/control/data/index/
   [junit4]   2> 2130384 T6958 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2130385 T6958 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/control/data/index
   [junit4]   2> 2130385 T6958 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2130391 T6958 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 2130391 T6958 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2130396 T6958 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2130397 T6958 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2130397 T6958 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2130398 T6958 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2130398 T6958 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2130399 T6958 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2130399 T6958 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2130400 T6958 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2130400 T6958 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2130401 T6958 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2130402 T6958 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2130402 T6958 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2130402 T6958 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2130403 T6958 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2130404 T6958 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2130405 T6958 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2130423 T6958 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2130428 T6958 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2130429 T6958 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2130429 T6958 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=44.14453125, floorSegmentMB=0.638671875, forceMergeDeletesPctAllowed=25.299205909566332, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8772573479417266
   [junit4]   2> 2130432 T6958 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 2130432 T6958 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2130433 T6958 oass.SolrIndexSearcher.<init> Opening Searcher@fc4eb68[collection1] main
   [junit4]   2> 2130433 T6958 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2130436 T6958 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2130437 T6958 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2130439 T6958 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2130439 T6958 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2130440 T6958 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2130442 T6958 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2130443 T6958 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2130443 T6958 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 2130444 T6958 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2130448 T6959 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@fc4eb68[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2130449 T6958 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2130452 T6962 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55834 collection:control_collection shard:shard1
   [junit4]   2> 2130453 T6933 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2130453 T6933 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2130455 T6962 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2130456 T6933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2130462 T6964 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@421f611c name:ZooKeeperConnection Watcher:127.0.0.1:55831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2130463 T6933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2130466 T6933 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2130475 T6933 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2130477 T6962 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2130483 T6962 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2130483 T6962 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1840 name=collection1 org.apache.solr.core.SolrCore@61caa3e6 url=http://127.0.0.1:55834/collection1 node=127.0.0.1:55834_ C1840_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:55834, core=collection1, node_name=127.0.0.1:55834_}
   [junit4]   2> 2130484 T6962 C1840 P55834 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55834/collection1/
   [junit4]   2> 2130485 T6962 C1840 P55834 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2130485 T6962 C1840 P55834 oasc.SyncStrategy.syncToMe http://127.0.0.1:55834/collection1/ has no replicas
   [junit4]   2> 2130485 T6962 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55834/collection1/ shard1
   [junit4]   2> 2130486 T6962 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2130485 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2130493 T6956 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2130507 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2130513 T6955 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2130517 T6964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2130535 T6956 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2130555 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2130673 T6964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2130679 T6955 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2130682 T6962 oasc.ZkController.register We are http://127.0.0.1:55834/collection1/ and leader is http://127.0.0.1:55834/collection1/
   [junit4]   2> 2130682 T6962 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55834
   [junit4]   2> 2130683 T6962 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2130683 T6962 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 2130683 T6962 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2130686 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2130687 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2130688 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2130689 T6962 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2130692 T6956 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2130693 T6956 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55834",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55834_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2130699 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2130806 T6955 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2130807 T6964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2132318 T6933 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2132320 T6933 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2132327 T6933 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55838
   [junit4]   2> 2132329 T6933 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2132329 T6933 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2132329 T6933 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-003
   [junit4]   2> 2132330 T6933 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-003/'
   [junit4]   2> 2132374 T6933 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-003/solr.xml
   [junit4]   2> 2132476 T6933 oasc.CoreContainer.<init> New CoreContainer 64550301
   [junit4]   2> 2132477 T6933 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-003/]
   [junit4]   2> 2132478 T6933 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2132478 T6933 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2132479 T6933 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2132479 T6933 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2132479 T6933 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2132479 T6933 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2132480 T6933 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2132480 T6933 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2132480 T6933 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2132481 T6933 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2132482 T6933 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2132482 T6933 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2132483 T6933 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55831/solr
   [junit4]   2> 2132483 T6933 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2132511 T6933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2132512 T6975 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18e51864 name:ZooKeeperConnection Watcher:127.0.0.1:55831 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2132514 T6933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2132520 T6933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2132526 T6977 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@120430f2 name:ZooKeeperConnection Watcher:127.0.0.1:55831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2132526 T6933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2132553 T6933 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2133565 T6933 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55838_
   [junit4]   2> 2133571 T6933 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55838_
   [junit4]   2> 2133587 T6964 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2133590 T6977 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2133591 T6955 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2133647 T6978 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2133648 T6978 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2133651 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2133651 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2133651 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2133651 T6978 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2133655 T6956 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2133657 T6956 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55838",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55838_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2133657 T6956 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 2133657 T6956 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 2133663 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2133664 T6955 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2133665 T6964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2133666 T6977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2134654 T6978 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2134654 T6978 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2134656 T6978 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2134656 T6978 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2134660 T6978 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2134660 T6978 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-003/collection1/'
   [junit4]   2> 2134663 T6978 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 2134663 T6978 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 2134664 T6978 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 2134772 T6978 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 2134854 T6978 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2134858 T6978 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2134868 T6978 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2135513 T6978 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2135517 T6978 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2135519 T6978 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2135526 T6978 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2135588 T6978 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2135588 T6978 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 2135589 T6978 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty1/
   [junit4]   2> 2135589 T6978 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@cc77302
   [junit4]   2> 2135590 T6978 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty1
   [junit4]   2> 2135591 T6978 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty1/index/
   [junit4]   2> 2135592 T6978 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 2135593 T6978 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty1/index
   [junit4]   2> 2135593 T6978 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2135600 T6978 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 2135601 T6978 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2135608 T6978 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2135608 T6978 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2135609 T6978 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2135609 T6978 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2135610 T6978 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2135610 T6978 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2135611 T6978 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2135611 T6978 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2135611 T6978 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2135612 T6978 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2135613 T6978 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2135613 T6978 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2135613 T6978 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2135614 T6978 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2135614 T6978 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2135615 T6978 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2135629 T6978 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2135633 T6978 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2135633 T6978 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2135634 T6978 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=44.14453125, floorSegmentMB=0.638671875, forceMergeDeletesPctAllowed=25.299205909566332, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8772573479417266
   [junit4]   2> 2135636 T6978 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 2135636 T6978 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2135637 T6978 oass.SolrIndexSearcher.<init> Opening Searcher@5d6e449a[collection1] main
   [junit4]   2> 2135637 T6978 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2135640 T6978 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2135640 T6978 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2135642 T6978 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2135642 T6978 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2135643 T6978 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2135645 T6978 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2135645 T6978 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2135645 T6978 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 2135646 T6978 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2135653 T6979 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5d6e449a[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2135657 T6978 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2135659 T6982 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55838 collection:collection1 shard:shard3
   [junit4]   2> 2135660 T6933 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2135661 T6933 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2135663 T6982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 2135678 T6982 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 2135682 T6982 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2135682 T6982 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1841 name=collection1 org.apache.solr.core.SolrCore@27af1e52 url=http://127.0.0.1:55838/collection1 node=127.0.0.1:55838_ C1841_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:55838, core=collection1, node_name=127.0.0.1:55838_}
   [junit4]   2> 2135683 T6982 C1841 P55838 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55838/collection1/
   [junit4]   2> 2135683 T6982 C1841 P55838 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2135683 T6982 C1841 P55838 oasc.SyncStrategy.syncToMe http://127.0.0.1:55838/collection1/ has no replicas
   [junit4]   2> 2135683 T6982 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55838/collection1/ shard3
   [junit4]   2> 2135684 T6982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 2135684 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2135694 T6956 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2135702 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2135708 T6955 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2135709 T6977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2135713 T6964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2135720 T6956 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2135725 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2135830 T6955 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2135830 T6964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2135831 T6977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2135861 T6982 oasc.ZkController.register We are http://127.0.0.1:55838/collection1/ and leader is http://127.0.0.1:55838/collection1/
   [junit4]   2> 2135862 T6982 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55838
   [junit4]   2> 2135862 T6982 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2135862 T6982 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2135865 T6982 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2135869 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2135870 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2135870 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2135872 T6982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2135877 T6956 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2135878 T6956 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55838",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55838_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2135884 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2135991 T6955 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2135991 T6977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2135991 T6964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2137280 T6933 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2137280 T6933 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2137284 T6933 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55841
   [junit4]   2> 2137286 T6933 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2137287 T6933 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2137287 T6933 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-004
   [junit4]   2> 2137288 T6933 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-004/'
   [junit4]   2> 2137334 T6933 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-004/solr.xml
   [junit4]   2> 2137460 T6933 oasc.CoreContainer.<init> New CoreContainer 926764590
   [junit4]   2> 2137460 T6933 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-004/]
   [junit4]   2> 2137462 T6933 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2137462 T6933 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2137462 T6933 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2137462 T6933 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2137463 T6933 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2137463 T6933 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2137463 T6933 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2137464 T6933 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2137464 T6933 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2137464 T6933 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2137465 T6933 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2137465 T6933 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2137466 T6933 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55831/solr
   [junit4]   2> 2137466 T6933 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2137468 T6933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2137474 T6993 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19d7ca57 name:ZooKeeperConnection Watcher:127.0.0.1:55831 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2137474 T6933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2137481 T6933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2137484 T6995 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5960b59 name:ZooKeeperConnection Watcher:127.0.0.1:55831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2137485 T6933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2137504 T6933 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2138515 T6933 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55841_
   [junit4]   2> 2138522 T6933 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55841_
   [junit4]   2> 2138542 T6955 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2138548 T6964 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2138549 T6977 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2138552 T6995 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2138592 T6996 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2138593 T6996 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2138595 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2138596 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2138596 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2138596 T6996 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2138600 T6956 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2138602 T6956 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55841",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55841_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2138603 T6956 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2138603 T6956 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2138611 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2138614 T6955 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2138615 T6995 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2138614 T6964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2138614 T6977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2139598 T6996 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2139598 T6996 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2139600 T6996 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2139600 T6996 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2139604 T6996 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2139604 T6996 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-004/collection1/'
   [junit4]   2> 2139606 T6996 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 2139607 T6996 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 2139607 T6996 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 2139691 T6996 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 2139777 T6996 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2139781 T6996 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2139798 T6996 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2140470 T6996 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2140474 T6996 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2140478 T6996 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2140485 T6996 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2140550 T6996 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2140550 T6996 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 2140551 T6996 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty2/
   [junit4]   2> 2140551 T6996 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@cc77302
   [junit4]   2> 2140552 T6996 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty2
   [junit4]   2> 2140552 T6996 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty2/index/
   [junit4]   2> 2140553 T6996 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 2140554 T6996 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty2/index
   [junit4]   2> 2140554 T6996 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2140557 T6996 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 2140558 T6996 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2140564 T6996 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2140564 T6996 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2140565 T6996 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2140565 T6996 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2140565 T6996 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2140566 T6996 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2140567 T6996 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2140567 T6996 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2140568 T6996 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2140568 T6996 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2140569 T6996 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2140569 T6996 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2140570 T6996 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2140570 T6996 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2140570 T6996 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2140571 T6996 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2140582 T6996 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2140587 T6996 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2140588 T6996 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2140588 T6996 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=44.14453125, floorSegmentMB=0.638671875, forceMergeDeletesPctAllowed=25.299205909566332, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8772573479417266
   [junit4]   2> 2140590 T6996 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 2140590 T6996 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2140591 T6996 oass.SolrIndexSearcher.<init> Opening Searcher@7576f622[collection1] main
   [junit4]   2> 2140591 T6996 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2140593 T6996 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2140594 T6996 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2140595 T6996 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2140595 T6996 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2140595 T6996 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2140596 T6996 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2140597 T6996 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2140597 T6996 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 2140597 T6996 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2140607 T6997 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7576f622[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2140610 T6996 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2140612 T7000 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55841 collection:collection1 shard:shard1
   [junit4]   2> 2140613 T6933 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2140613 T6933 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2140614 T7000 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2140627 T7000 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2140631 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2140636 T7000 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2140636 T7000 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1842 name=collection1 org.apache.solr.core.SolrCore@35e3ae2a url=http://127.0.0.1:55841/collection1 node=127.0.0.1:55841_ C1842_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:55841, core=collection1, node_name=127.0.0.1:55841_}
   [junit4]   2> 2140636 T7000 C1842 P55841 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55841/collection1/
   [junit4]   2> 2140636 T7000 C1842 P55841 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2140637 T7000 C1842 P55841 oasc.SyncStrategy.syncToMe http://127.0.0.1:55841/collection1/ has no replicas
   [junit4]   2> 2140637 T7000 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55841/collection1/ shard1
   [junit4]   2> 2140638 T7000 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2140638 T6956 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2140645 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2140648 T6977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2140648 T6964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2140649 T6955 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2140651 T6995 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2140663 T6956 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2140669 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2140775 T6964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2140775 T6995 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2140776 T6955 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2140775 T6977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2140816 T7000 oasc.ZkController.register We are http://127.0.0.1:55841/collection1/ and leader is http://127.0.0.1:55841/collection1/
   [junit4]   2> 2140816 T7000 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55841
   [junit4]   2> 2140816 T7000 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2140816 T7000 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2140817 T7000 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2140821 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2140821 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2140821 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2140824 T7000 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2140825 T6956 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2140828 T6956 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55841",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:55841_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 2140835 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2140941 T6977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2140941 T6995 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2140942 T6964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2140942 T6955 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2142255 T6933 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 2142256 T6933 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2142260 T6933 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55844
   [junit4]   2> 2142262 T6933 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2142263 T6933 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2142263 T6933 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-005
   [junit4]   2> 2142263 T6933 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-005/'
   [junit4]   2> 2142324 T6933 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-005/solr.xml
   [junit4]   2> 2142423 T6933 oasc.CoreContainer.<init> New CoreContainer 1666291875
   [junit4]   2> 2142424 T6933 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-005/]
   [junit4]   2> 2142426 T6933 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2142426 T6933 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2142426 T6933 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2142427 T6933 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2142427 T6933 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2142428 T6933 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2142428 T6933 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2142428 T6933 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2142429 T6933 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2142429 T6933 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2142430 T6933 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2142430 T6933 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2142430 T6933 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55831/solr
   [junit4]   2> 2142431 T6933 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2142433 T6933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2142439 T7011 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fc2846 name:ZooKeeperConnection Watcher:127.0.0.1:55831 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2142439 T6933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2142444 T6933 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2142448 T7013 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@619163ca name:ZooKeeperConnection Watcher:127.0.0.1:55831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2142449 T6933 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2142473 T6933 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2143485 T6933 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55844_
   [junit4]   2> 2143490 T6933 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55844_
   [junit4]   2> 2143506 T6995 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2143507 T6964 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2143507 T7013 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2143510 T6977 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2143510 T6955 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2143544 T7014 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2143544 T7014 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2143549 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2143549 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2143550 T6955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenCh

[...truncated too long message...]

_size=6586,transaction_logs_total_number=2}
   [junit4]   2> 2214781 T7198 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 2214781 T7198 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 2214781 T7198 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 2214783 T7198 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 2214787 T7198 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 2214787 T7198 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty12/index.20140531111827370 [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty12/index.20140531111827370;done=false>>]
   [junit4]   2> 2214787 T7198 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty12/index.20140531111827370
   [junit4]   2> 2214787 T7198 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty12 [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty12;done=false>>]
   [junit4]   2> 2214788 T7198 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001/tempDir-001/jetty12
   [junit4]   2> 2214788 T7198 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2214789 T7198 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-6FEB1EEAFFA9C377-001
   [junit4]   2> NOTE: test params are: codec=Lucene3x, sim=DefaultSimilarity, locale=ar_DZ, timezone=America/Coral_Harbour
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_55 (64-bit)/cpus=2,threads=1,free=75403000,total=349831168
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsSyncSliceTest, TestTrie, PrimUtilsTest, ZkControllerTest, BlockDirectoryTest, UpdateRequestProcessorFactoryTest, DefaultValueUpdateProcessorTest, DistributedDebugComponentTest, CustomCollectionTest, TestManagedResource, TestFuzzyAnalyzedSuggestions, DirectUpdateHandlerTest, TestSolrDeletionPolicy2, TestOverriddenPrefixQueryForCustomFieldType, SolrCoreCheckLockOnStartupTest, TestDocSet, TestJmxMonitoredMap, TestWriterPerf, EnumFieldTest, ChangedSchemaMergeTest, OutputWriterTest, RequiredFieldsTest, TestExceedMaxTermLength, TestValueSourceCache, TestSolrXMLSerializer, TestNRTOpen, TestStressUserVersions, TestPhraseSuggestions, TestGroupingSearch, TestStressRecovery, BinaryUpdateRequestHandlerTest, MergeStrategyTest, TestFieldTypeResource, TestCoreContainer, TestShardHandlerFactory, TestSearcherReuse, CacheHeaderTest, TestSolrDeletionPolicy1, SpellCheckComponentTest, DistributedTermsComponentTest, ShardRoutingTest, MultiThreadedOCPTest, TestFunctionQuery, PreAnalyzedFieldTest, ZkNodePropsTest, FileBasedSpellCheckerTest, TestJmxIntegration, SpellCheckCollatorTest, TestSort, EchoParamsTest, LeaderElectionTest, SolrIndexConfigTest, NotRequiredUniqueKeyTest, BadComponentTest, TestSuggestSpellingConverter, TestBlendedInfixSuggestions, TestQueryUtils, HdfsDirectoryTest, TestLMDirichletSimilarityFactory, TestRangeQuery, TermVectorComponentDistributedTest, JSONWriterTest, SimplePostToolTest, TestNoOpRegenerator, MBeansHandlerTest, ShardRoutingCustomTest, TestFastLRUCache, TestCharFilters, TestDistribDocBasedVersion, BlockCacheTest, TestJoin, ConnectionManagerTest, SOLR749Test, TestAnalyzeInfixSuggestions, TestDefaultSimilarityFactory, OpenExchangeRatesOrgProviderTest, TestPHPSerializedResponseWriter, PeerSyncTest, TestSerializedLuceneMatchVersion, DistributedExpandComponentTest, SignatureUpdateProcessorFactoryTest, TestReload, RecoveryZkTest, TestReplicationHandler, InfoHandlerTest, UnloadDistributedZkTest, TestLMJelinekMercerSimilarityFactory, TestAtomicUpdateErrorCases, TestRandomMergePolicy, FieldMutatingUpdateProcessorTest, XmlUpdateRequestHandlerTest, FileUtilsTest, TestUniqueKeyFieldResource, TestLuceneMatchVersion, TestDFRSimilarityFactory, AnalysisErrorHandlingTest, DistributedSuggestComponentTest, DOMUtilTest, URLClassifyProcessorTest, SolrInfoMBeanTest, TestPartialUpdateDeduplication, TestRequestStatusCollectionAPI, SolrPluginUtilsTest, TestFieldSortValues, TestReloadAndDeleteDocs, CSVRequestHandlerTest, CachingDirectoryFactoryTest, TestXIncludeConfig, DirectSolrSpellCheckerTest, SearchHandlerTest, CoreAdminCreateDiscoverTest, TestDocumentBuilder, TestRecovery, OverseerCollectionProcessorTest, TestIBSimilarityFactory, DateFieldTest, TestPostingsSolrHighlighter, FastVectorHighlighterTest, TestDistributedMissingSort, TestHighFrequencyDictionaryFactory, SolrXmlInZkTest, AliasIntegrationTest, TestRestManager, DeleteReplicaTest, TestCloudManagedSchema, TestStressVersions, DocExpirationUpdateProcessorFactoryTest, HdfsUnloadDistributedZkTest, ScriptEngineTest, LeaderElectionIntegrationTest, SolrTestCaseJ4Test, SynonymTokenizerTest, TestSolrXmlPersistor, LegacyHTMLStripCharFilterTest, MultiTermTest, HdfsCollectionsAPIDistributedZkTest, SpellingQueryConverterTest, TestElisionMultitermQuery, TestCustomSort, BasicDistributedZk2Test, SuggesterTest, BJQParserTest, ClusterStateUpdateTest, DocValuesTest, TestManagedSchema, SolrRequestParserTest, TestStandardQParsers, TestSolr4Spatial, HighlighterTest, JsonLoaderTest, RemoteQueryErrorTest, DeleteShardTest, TestSolrQueryParserResource, FullSolrCloudDistribCmdsTest, TestMiniSolrCloudCluster, AutoCommitTest, TestCloudInspectUtil, TestHashPartitioner, ParsingFieldUpdateProcessorsTest, SpellPossibilityIteratorTest, CursorPagingTest, TestBinaryResponseWriter, DistributedQueryComponentCustomSortTest, TestManagedStopFilterFactory, ConvertedLegacyTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestSchemaNameResource, HdfsChaosMonkeySafeLeaderTest, SyncSliceTest, ExternalFileFieldSortTest, TestDistributedGrouping, BadIndexSchemaTest, TestInitQParser, SortByFunctionTest, TestUpdate, PingRequestHandlerTest, PluginInfoTest, DirectUpdateHandlerOptimizeTest, WordBreakSolrSpellCheckerTest, MinimalSchemaTest, TestSchemaSimilarityResource, TestBadConfig, DateMathParserTest, HdfsLockFactoryTest, CollectionsAPIAsyncDistributedZkTest, QueryElevationComponentTest, TestCollationKeyRangeQueries, TestDefaultSearchFieldResource, SampleTest, OverseerStatusTest, TermsComponentTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest]
   [junit4] Completed in 88.82s, 1 test, 1 error <<< FAILURES!

[...truncated 615 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1297: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:921: There were test failures: 396 suites, 1640 tests, 1 error, 41 ignored (14 assumptions)

Total time: 113 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure