You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/05/07 15:54:59 UTC

[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.8.0) - Build # 1559 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1559/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

All tests passed

Build Log:
[...truncated 20421 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:92: The following files contain @author tags, tabs or nocommits:
* solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java

Total time: 103 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1563 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1563/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseG1GC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
shard1 is not consistent.  Got 102 from http://127.0.0.1:51324/collection1lastClient and got 94 from http://127.0.0.1:51333/collection1

Stack Trace:
java.lang.AssertionError: shard1 is not consistent.  Got 102 from http://127.0.0.1:51324/collection1lastClient and got 94 from http://127.0.0.1:51333/collection1
	at __randomizedtesting.SeedInfo.seed([E7C2A15175B9AA9D:66242F4902E6CAA1]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1130)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1109)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11176 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/init-core-data-001
   [junit4]   2> 2034278 T6715 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 2034279 T6715 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 2034294 T6715 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 2034295 T6715 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2034298 T6716 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2034399 T6715 oasc.ZkTestServer.run start zk server on port:51317
   [junit4]   2> 2034401 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2034411 T6722 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74d50c0b name:ZooKeeperConnection Watcher:127.0.0.1:51317 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2034412 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2034413 T6715 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2034430 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2034437 T6724 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a100f8b name:ZooKeeperConnection Watcher:127.0.0.1:51317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2034439 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2034440 T6715 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2034451 T6715 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2034459 T6715 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2034465 T6715 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2034474 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2034475 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2034490 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 2034491 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2034498 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2034498 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2034505 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2034505 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2034512 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2034513 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2034520 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2034521 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2034527 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2034527 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2034537 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2034537 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2034546 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2034547 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2034557 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2034558 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2034567 T6715 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2034568 T6715 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2036217 T6715 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2036223 T6715 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
   [junit4]   2> 2036224 T6715 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2036225 T6715 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2036225 T6715 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002
   [junit4]   2> 2036226 T6715 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002/'
   [junit4]   2> 2036294 T6715 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002/solr.xml
   [junit4]   2> 2036617 T6715 oasc.CoreContainer.<init> New CoreContainer 1161568699
   [junit4]   2> 2036624 T6715 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002/]
   [junit4]   2> 2036638 T6715 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2036640 T6715 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2036646 T6715 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2036647 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2036649 T6715 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2036651 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2036657 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2036658 T6715 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2036660 T6715 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2036672 T6715 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2036675 T6715 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2036679 T6715 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2036683 T6715 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51317/solr
   [junit4]   2> 2036685 T6715 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2036710 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2036718 T6735 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6eca9f7f name:ZooKeeperConnection Watcher:127.0.0.1:51317 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2036718 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2036724 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2036728 T6737 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e771953 name:ZooKeeperConnection Watcher:127.0.0.1:51317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2036729 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2036733 T6715 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2036743 T6715 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2036753 T6715 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2036763 T6715 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2036774 T6715 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2036786 T6715 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2036791 T6715 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51320_
   [junit4]   2> 2036794 T6715 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51320_
   [junit4]   2> 2036805 T6715 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2036814 T6715 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2036827 T6715 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51320_
   [junit4]   2> 2036827 T6715 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2036835 T6715 oasc.Overseer.start Overseer (id=91723437819297795-127.0.0.1:51320_-n_0000000000) starting
   [junit4]   2> 2036846 T6715 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2036884 T6739 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2036888 T6739 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:51320_
   [junit4]   2> 2036889 T6715 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2036897 T6715 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2036903 T6715 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2036912 T6738 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2036924 T6740 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 2036925 T6740 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2036928 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2036928 T6740 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2036933 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2036935 T6738 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51320",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51320_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2036936 T6738 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 2036936 T6738 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2036943 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2036946 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2037930 T6740 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2037930 T6740 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 2037932 T6740 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2037933 T6740 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2037936 T6740 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2037936 T6740 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002/collection1/'
   [junit4]   2> 2037938 T6740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 2037938 T6740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 2037940 T6740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 2038071 T6740 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 2038217 T6740 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2038221 T6740 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2038237 T6740 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2039080 T6740 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2039086 T6740 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2039090 T6740 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2039100 T6740 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2039197 T6740 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 2039197 T6740 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 2039199 T6740 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/control/data/
   [junit4]   2> 2039199 T6740 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@57a9514b
   [junit4]   2> 2039201 T6740 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/control/data
   [junit4]   2> 2039203 T6740 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/control/data/index/
   [junit4]   2> 2039204 T6740 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2039205 T6740 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/control/data/index
   [junit4]   2> 2039205 T6740 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=54.892578125, floorSegmentMB=1.830078125, forceMergeDeletesPctAllowed=3.2255022366820043, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2039212 T6740 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 2039212 T6740 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2039223 T6740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2039224 T6740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2039225 T6740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2039225 T6740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2039225 T6740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2039226 T6740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2039227 T6740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2039227 T6740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2039227 T6740 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2039228 T6740 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2039229 T6740 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2039229 T6740 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2039230 T6740 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2039231 T6740 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2039232 T6740 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2039232 T6740 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2039234 T6740 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2039248 T6740 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2039254 T6740 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2039254 T6740 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2039256 T6740 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=27, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 2039259 T6740 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 2039260 T6740 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2039261 T6740 oass.SolrIndexSearcher.<init> Opening Searcher@723c43a9[collection1] main
   [junit4]   2> 2039261 T6740 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2039269 T6740 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2039270 T6740 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2039271 T6740 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2039272 T6740 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2039272 T6740 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2039274 T6740 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2039276 T6740 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2039276 T6740 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 2039276 T6740 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2039286 T6741 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@723c43a9[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2039289 T6740 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2039292 T6744 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51320 collection:control_collection shard:shard1
   [junit4]   2> 2039293 T6715 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2039294 T6715 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2039294 T6744 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2039298 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2039304 T6746 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2308dacb name:ZooKeeperConnection Watcher:127.0.0.1:51317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2039305 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2039309 T6715 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2039317 T6715 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2039320 T6744 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2039326 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2039330 T6744 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2039330 T6744 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1008 name=collection1 org.apache.solr.core.SolrCore@502de817 url=http://127.0.0.1:51320/collection1 node=127.0.0.1:51320_ C1008_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:51320, core=collection1, node_name=127.0.0.1:51320_}
   [junit4]   2> 2039331 T6744 C1008 P51320 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51320/collection1/
   [junit4]   2> 2039331 T6744 C1008 P51320 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2039332 T6744 C1008 P51320 oasc.SyncStrategy.syncToMe http://127.0.0.1:51320/collection1/ has no replicas
   [junit4]   2> 2039332 T6744 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51320/collection1/ shard1
   [junit4]   2> 2039332 T6744 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2039333 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2039350 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2039354 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2039355 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2039377 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2039388 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2039495 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2039495 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2039528 T6744 oasc.ZkController.register We are http://127.0.0.1:51320/collection1/ and leader is http://127.0.0.1:51320/collection1/
   [junit4]   2> 2039529 T6744 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51320
   [junit4]   2> 2039530 T6744 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2039530 T6744 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 2039531 T6744 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2039537 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2039537 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2039539 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2039542 T6744 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2039547 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2039548 T6738 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51320",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51320_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2039558 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2039667 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2039669 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2041187 T6715 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2041189 T6715 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2041196 T6715 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
   [junit4]   2> 2041196 T6715 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2041197 T6715 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2041197 T6715 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003
   [junit4]   2> 2041197 T6715 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003/'
   [junit4]   2> 2041258 T6715 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003/solr.xml
   [junit4]   2> 2041375 T6715 oasc.CoreContainer.<init> New CoreContainer 1081531971
   [junit4]   2> 2041376 T6715 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003/]
   [junit4]   2> 2041377 T6715 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2041378 T6715 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2041378 T6715 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2041378 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2041379 T6715 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2041379 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2041380 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2041380 T6715 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2041380 T6715 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2041381 T6715 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2041382 T6715 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2041383 T6715 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2041383 T6715 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51317/solr
   [junit4]   2> 2041384 T6715 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2041385 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2041391 T6757 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3bfa21c5 name:ZooKeeperConnection Watcher:127.0.0.1:51317 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2041391 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2041397 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2041400 T6759 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@594dcf55 name:ZooKeeperConnection Watcher:127.0.0.1:51317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2041400 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2041425 T6715 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2042435 T6715 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51324_
   [junit4]   2> 2042438 T6715 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51324_
   [junit4]   2> 2042477 T6760 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2042477 T6760 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2042480 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2042480 T6760 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2042481 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2042481 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2042485 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2042487 T6738 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51324",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51324_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2042487 T6738 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 2042487 T6738 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2042494 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2042496 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2042497 T6759 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2042497 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2043482 T6760 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2043482 T6760 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2043484 T6760 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2043484 T6760 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2043487 T6760 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2043487 T6760 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003/collection1/'
   [junit4]   2> 2043489 T6760 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 2043490 T6760 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 2043491 T6760 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 2043594 T6760 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 2043744 T6760 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2043747 T6760 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2043758 T6760 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2044419 T6760 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2044423 T6760 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2044425 T6760 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2044433 T6760 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2044492 T6760 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2044492 T6760 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 2044494 T6760 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty1/
   [junit4]   2> 2044494 T6760 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@57a9514b
   [junit4]   2> 2044496 T6760 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty1
   [junit4]   2> 2044497 T6760 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty1/index/
   [junit4]   2> 2044497 T6760 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 2044498 T6760 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty1/index
   [junit4]   2> 2044498 T6760 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=54.892578125, floorSegmentMB=1.830078125, forceMergeDeletesPctAllowed=3.2255022366820043, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2044504 T6760 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 2044504 T6760 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2044514 T6760 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2044514 T6760 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2044515 T6760 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2044515 T6760 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2044515 T6760 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2044515 T6760 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2044516 T6760 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2044517 T6760 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2044517 T6760 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2044518 T6760 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2044519 T6760 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2044520 T6760 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2044520 T6760 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2044521 T6760 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2044522 T6760 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2044523 T6760 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2044524 T6760 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2044537 T6760 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2044542 T6760 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2044543 T6760 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2044544 T6760 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=27, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 2044546 T6760 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 2044546 T6760 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2044547 T6760 oass.SolrIndexSearcher.<init> Opening Searcher@6ad4dfe1[collection1] main
   [junit4]   2> 2044547 T6760 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2044550 T6760 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2044550 T6760 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2044552 T6760 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2044552 T6760 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2044553 T6760 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2044554 T6760 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2044555 T6760 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2044555 T6760 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 2044555 T6760 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2044561 T6761 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6ad4dfe1[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2044565 T6760 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2044568 T6764 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51324 collection:collection1 shard:shard1
   [junit4]   2> 2044569 T6715 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2044569 T6715 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2044571 T6764 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2044582 T6764 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2044588 T6764 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2044589 T6764 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1009 name=collection1 org.apache.solr.core.SolrCore@1665b87c url=http://127.0.0.1:51324/collection1 node=127.0.0.1:51324_ C1009_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:51324, core=collection1, node_name=127.0.0.1:51324_}
   [junit4]   2> 2044589 T6764 C1009 P51324 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51324/collection1/
   [junit4]   2> 2044590 T6764 C1009 P51324 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2044590 T6764 C1009 P51324 oasc.SyncStrategy.syncToMe http://127.0.0.1:51324/collection1/ has no replicas
   [junit4]   2> 2044590 T6764 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51324/collection1/ shard1
   [junit4]   2> 2044591 T6764 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2044595 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2044605 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2044610 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2044624 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2044625 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2044629 T6759 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2044648 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2044768 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2044769 T6759 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2044776 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2044821 T6764 oasc.ZkController.register We are http://127.0.0.1:51324/collection1/ and leader is http://127.0.0.1:51324/collection1/
   [junit4]   2> 2044822 T6764 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51324
   [junit4]   2> 2044822 T6764 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2044822 T6764 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2044822 T6764 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2044826 T6764 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2044828 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2044828 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2044828 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2044832 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2044834 T6738 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51324",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51324_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2044839 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2044946 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2044947 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2044952 T6759 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2046199 T6715 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2046200 T6715 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2046206 T6715 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
   [junit4]   2> 2046207 T6715 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2046207 T6715 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2046208 T6715 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004
   [junit4]   2> 2046208 T6715 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004/'
   [junit4]   2> 2046264 T6715 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004/solr.xml
   [junit4]   2> 2046359 T6715 oasc.CoreContainer.<init> New CoreContainer 1835826604
   [junit4]   2> 2046360 T6715 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004/]
   [junit4]   2> 2046362 T6715 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2046362 T6715 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2046362 T6715 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2046363 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2046363 T6715 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2046363 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2046364 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2046364 T6715 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2046364 T6715 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2046365 T6715 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2046366 T6715 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2046367 T6715 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2046367 T6715 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51317/solr
   [junit4]   2> 2046367 T6715 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2046369 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2046372 T6775 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6daedf53 name:ZooKeeperConnection Watcher:127.0.0.1:51317 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2046373 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2046378 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2046381 T6777 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b1040d1 name:ZooKeeperConnection Watcher:127.0.0.1:51317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2046381 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2046401 T6715 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2047411 T6715 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51327_
   [junit4]   2> 2047414 T6715 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51327_
   [junit4]   2> 2047438 T6778 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2047439 T6778 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2047441 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2047441 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2047441 T6778 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2047442 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2047446 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2047447 T6738 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51327",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51327_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2047448 T6738 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 2047448 T6738 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 2047454 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2047456 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2047456 T6759 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2047456 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2047456 T6777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2048444 T6778 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2048444 T6778 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2048446 T6778 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2048446 T6778 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2048450 T6778 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2048451 T6778 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004/collection1/'
   [junit4]   2> 2048452 T6778 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 2048452 T6778 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 2048453 T6778 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 2048541 T6778 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 2048634 T6778 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2048637 T6778 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2048649 T6778 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2049375 T6778 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2049383 T6778 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2049388 T6778 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2049397 T6778 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2049457 T6778 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2049458 T6778 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 2049459 T6778 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty2/
   [junit4]   2> 2049459 T6778 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@57a9514b
   [junit4]   2> 2049461 T6778 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty2
   [junit4]   2> 2049462 T6778 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty2/index/
   [junit4]   2> 2049462 T6778 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 2049463 T6778 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty2/index
   [junit4]   2> 2049465 T6778 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=10, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=54.892578125, floorSegmentMB=1.830078125, forceMergeDeletesPctAllowed=3.2255022366820043, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2049471 T6778 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 2049472 T6778 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2049480 T6778 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2049481 T6778 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2049482 T6778 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2049482 T6778 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2049482 T6778 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2049482 T6778 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2049483 T6778 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2049484 T6778 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2049484 T6778 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2049486 T6778 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2049486 T6778 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2049487 T6778 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2049487 T6778 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2049488 T6778 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2049488 T6778 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2049489 T6778 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2049490 T6778 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2049503 T6778 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2049508 T6778 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2049509 T6778 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2049510 T6778 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=27, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 2049512 T6778 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 2049512 T6778 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2049512 T6778 oass.SolrIndexSearcher.<init> Opening Searcher@1852ba3f[collection1] main
   [junit4]   2> 2049513 T6778 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2049516 T6778 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2049517 T6778 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2049518 T6778 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2049518 T6778 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2049519 T6778 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2049521 T6778 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2049521 T6778 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2049521 T6778 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 2049522 T6778 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2049530 T6779 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1852ba3f[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2049533 T6778 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2049535 T6782 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51327 collection:collection1 shard:shard3
   [junit4]   2> 2049536 T6715 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2049537 T6715 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2049538 T6782 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 2049548 T6782 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 2049555 T6782 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2049555 T6782 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1010 name=collection1 org.apache.solr.core.SolrCore@72ae959e url=http://127.0.0.1:51327/collection1 node=127.0.0.1:51327_ C1010_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:51327, core=collection1, node_name=127.0.0.1:51327_}
   [junit4]   2> 2049555 T6782 C1010 P51327 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51327/collection1/
   [junit4]   2> 2049556 T6782 C1010 P51327 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2049556 T6782 C1010 P51327 oasc.SyncStrategy.syncToMe http://127.0.0.1:51327/collection1/ has no replicas
   [junit4]   2> 2049556 T6782 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51327/collection1/ shard3
   [junit4]   2> 2049556 T6782 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 2049558 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2049564 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2049574 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2049577 T6759 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2049577 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2049578 T6777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2049578 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2049591 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2049599 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2049707 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2049707 T6759 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2049707 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2049707 T6777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2049738 T6782 oasc.ZkController.register We are http://127.0.0.1:51327/collection1/ and leader is http://127.0.0.1:51327/collection1/
   [junit4]   2> 2049739 T6782 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51327
   [junit4]   2> 2049739 T6782 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2049740 T6782 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2049740 T6782 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2049745 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2049746 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2049746 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2049748 T6782 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2049752 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2049753 T6738 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51327",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51327_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 2049760 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2049865 T6737 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2049866 T6777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2049865 T6759 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2049866 T6746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2051208 T6715 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 2051209 T6715 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2051214 T6715 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:?????
   [junit4]   2> 2051215 T6715 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2051215 T6715 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2051216 T6715 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-005
   [junit4]   2> 2051216 T6715 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-005/'
   [junit4]   2> 2051277 T6715 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-005/solr.xml
   [junit4]   2> 2051377 T6715 oasc.CoreContainer.<init> New CoreContainer 2009942148
   [junit4]   2> 2051377 T6715 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-005/]
   [junit4]   2> 2051379 T6715 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2051379 T6715 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2051380 T6715 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2051380 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2051380 T6715 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2051381 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2051381 T6715 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2051381 T6715 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2051381 T6715 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2051382 T6715 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2051383 T6715 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2051383 T6715 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2051383 T6715 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51317/solr
   [junit4]   2> 2051384 T6715 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2051386 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2051389 T6793 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46540f8e name:ZooKeeperConnection Watcher:127.0.0.1:51317 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2051390 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2051394 T6715 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2051397 T6795 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9b7bf39 name:ZooKeeperConnection Watcher:127.0.0.1:51317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2051397 T6715 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2051416 T6715 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2052426 T6715 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51330_
   [junit4]   2> 2052428 T6715 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51330_
   [junit4]   2> 2052453 T6796 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2052454 T6796 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2052456 T6796 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2052456 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2052457 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2052457 T6737 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2052460 T6738 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2052462 T6738 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51330",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51330_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2052462 T6738 oasc.Overseer$ClusterStateUpdater.updateState Collection already exi

[...truncated too long message...]

tate Closing SolrCoreState
   [junit4]   2> 2149040 T6959 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 2149040 T6959 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 2149064 T6959 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 2149066 T6959 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 2149066 T6959 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty11 [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty11;done=false>>]
   [junit4]   2> 2149066 T6959 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty11
   [junit4]   2> 2149067 T6959 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty11/index [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty11/index;done=false>>]
   [junit4]   2> 2149067 T6959 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001/tempDir-001/jetty11/index
   [junit4]   2> 2149067 T6959 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 2149068 T6959 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-E7C2A15175B9AA9D-001
   [junit4]   2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=461), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=461)), sim=DefaultSimilarity, locale=hi_IN, timezone=Africa/Sao_Tome
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_55 (64-bit)/cpus=2,threads=1,free=379610632,total=534773760
   [junit4]   2> NOTE: All tests run in this JVM: [TestPseudoReturnFields, TestDistributedSearch, SynonymTokenizerTest, DirectUpdateHandlerTest, TestCollectionAPI, TestComplexPhraseQParserPlugin, StatelessScriptUpdateProcessorFactoryTest, HdfsSyncSliceTest, NotRequiredUniqueKeyTest, EnumFieldTest, TestDynamicFieldCollectionResource, TestSolrXmlPersistor, TestQueryUtils, HdfsCollectionsAPIDistributedZkTest, TestSolrXMLSerializer, TestConfigSets, PreAnalyzedUpdateProcessorTest, TestSearcherReuse, TestPHPSerializedResponseWriter, TestPhraseSuggestions, TestBinaryField, TestInitQParser, TestLMDirichletSimilarityFactory, TestReplicationHandler, HttpPartitionTest, TestShortCircuitedRequests, PrimitiveFieldTypeTest, DistributedExpandComponentTest, TestDocBasedVersionConstraints, TestJmxMonitoredMap, CursorMarkTest, TestSchemaSimilarityResource, QueryElevationComponentTest, JsonLoaderTest, LoggingHandlerTest, ZkNodePropsTest, TestSearchPerf, OverseerRolesTest, TestDistribDocBasedVersion, MoreLikeThisHandlerTest, FieldAnalysisRequestHandlerTest, TestInfoStreamLogging, RangeFacetTest, TestRealTimeGet, SliceStateUpdateTest, StatsComponentTest, TestTrie, BadCopyFieldTest, TestCollationFieldDocValues, LeaderElectionIntegrationTest, TestModifyConfFiles, TestRTGBase, SolrCoreTest, DisMaxRequestHandlerTest, ClusterStateUpdateTest, ChangedSchemaMergeTest, LegacyHTMLStripCharFilterTest, DistributedQueryComponentCustomSortTest, ChaosMonkeyNothingIsSafeTest, ConvertedLegacyTest, LeaderElectionTest, BasicDistributedZk2Test, TestCloudInspectUtil, TestImplicitCoreProperties, AbstractAnalyticsStatsTest, TestFoldingMultitermQuery, ExpressionTest, CacheHeaderTest, SearchHandlerTest, SolrPluginUtilsTest, TestSolr4Spatial, TestRestManager, NoFacetTest, TestDefaultSearchFieldResource, TestUtils, SampleTest, TestLazyCores, PingRequestHandlerTest, SSLMigrationTest, TestManagedSynonymFilterFactory, TestOverriddenPrefixQueryForCustomFieldType, SuggestComponentTest, PreAnalyzedFieldTest, RemoteQueryErrorTest, BadComponentTest, RegexBoostProcessorTest, TestSort, TestSolrDeletionPolicy2, WordBreakSolrSpellCheckerTest, TestElisionMultitermQuery, TestSchemaResource, TestPostingsSolrHighlighter, FunctionTest, BasicDistributedZkTest, TestCollapseQParserPlugin, TestOmitPositions, TestShardHandlerFactory, TestNRTOpen, TestZkChroot, TestFastOutputStream, SuggesterTest, TestIndexSearcher, ShardRoutingCustomTest, TestSolrXml, ParsingFieldUpdateProcessorsTest, AbstractAnalyticsFacetTest, TestSolrJ, SuggesterFSTTest, TestQuerySenderNoQuery, TestAddFieldRealTimeGet, TestFieldResource, TestSchemaVersionResource, ExternalFileFieldSortTest, FieldFacetTest, DateMathParserTest, CustomCollectionTest, SchemaVersionSpecificBehaviorTest, UpdateParamsTest, TestDistributedGrouping, MBeansHandlerTest, TestQuerySenderListener, TestFaceting, UpdateRequestProcessorFactoryTest, PathHierarchyTokenizerFactoryTest, XsltUpdateRequestHandlerTest, QueryFacetTest, SuggesterWFSTTest, RequestHandlersTest, TestMultiCoreConfBootstrap, CurrencyFieldXmlFileTest, TestDynamicFieldResource, SoftAutoCommitTest, SpellCheckCollatorTest, TestJmxIntegration, TestDefaultSimilarityFactory, ConnectionManagerTest, TestBadConfig, DocExpirationUpdateProcessorFactoryTest, CursorPagingTest, DebugComponentTest, SolrRequestParserTest, TestSimpleQParserPlugin, SyncSliceTest, OverseerCollectionProcessorTest, TestCursorMarkWithoutUniqueKey, BasicFunctionalityTest, ModifyConfFileTest, HdfsBasicDistributedZkTest, TermsComponentTest, HdfsRecoveryZkTest, OpenCloseCoreStressTest, TestFieldTypeResource, UniqFieldsUpdateProcessorFactoryTest, TestFastWriter, OverseerTest, AddBlockUpdateTest, TestComponentsName, TestExceedMaxTermLength, DistanceFunctionTest, DocumentBuilderTest, QueryResultKeyTest, TestSolrDeletionPolicy1, DistributedSpellCheckComponentTest, SolrTestCaseJ4Test, TestSolrXmlPersistence, TestSuggestSpellingConverter, DistribDocExpirationUpdateProcessorTest, TestManagedResource, ReturnFieldsTest, ZkCLITest, AnalysisAfterCoreReloadTest, TestMergePolicyConfig, TestFieldSortValues, TestReloadAndDeleteDocs, TestRandomMergePolicy, TestCSVLoader, TestFuzzyAnalyzedSuggestions, TestSurroundQueryParser, AlternateDirectoryTest, BlockDirectoryTest, SolrInfoMBeanTest, TestLRUCache, SpellPossibilityIteratorTest, PrimUtilsTest, CoreAdminHandlerTest, StressHdfsTest, TestSolrQueryParserDefaultOperatorResource, TestCopyFieldCollectionResource, SimpleFacetsTest, TestIndexingPerformance, BlockCacheTest, ScriptEngineTest, ChaosMonkeySafeLeaderTest]
   [junit4] Completed in 115.85s, 1 test, 1 failure <<< FAILURES!

[...truncated 618 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1296: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:920: There were test failures: 399 suites, 1674 tests, 1 failure, 42 ignored (15 assumptions)

Total time: 117 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.8.0) - Build # 1562 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1562/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 20550 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:92: The following files contain @author tags, tabs or nocommits:
* solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java

Total time: 102 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1561 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1561/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

All tests passed

Build Log:
[...truncated 17656 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-map-reduce/test/temp/junit4-J0-20140508_055916_990.syserr
   [junit4] >>> JVM J0: stderr (verbatim) ----
   [junit4] 2014-05-08 05:59:30.131 java[559:650b] Unable to load realm info from SCDynamicStore
   [junit4] <<< JVM J0: EOF ----

[...truncated 2849 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:92: The following files contain @author tags, tabs or nocommits:
* solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java

Total time: 127 minutes 24 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1560 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1560/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 17666 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-map-reduce/test/temp/junit4-J0-20140507_221254_942.syserr
   [junit4] >>> JVM J0: stderr (verbatim) ----
   [junit4] 2014-05-07 22:13:50.647 java[277:6d03] Unable to load realm info from SCDynamicStore
   [junit4] <<< JVM J0: EOF ----

[...truncated 2849 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:92: The following files contain @author tags, tabs or nocommits:
* solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java

Total time: 127 minutes 11 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure