You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2015/03/24 12:27:04 UTC

[JENKINS] Lucene-Solr-5.x-Linux (64bit/jdk1.7.0_76) - Build # 11903 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11903/
Java: 64bit/jdk1.7.0_76 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.handler.TestSolrConfigHandlerCloud.test

Error Message:
Could not get expected value  'CY val modified' for path 'response/params/y/c' full output: {   "responseHeader":{     "status":0,     "QTime":0},   "response":{     "znodeVersion":0,     "params":{"x":{         "a":"A val",         "b":"B val",         "":{"v":0}}}}}

Stack Trace:
java.lang.AssertionError: Could not get expected value  'CY val modified' for path 'response/params/y/c' full output: {
  "responseHeader":{
    "status":0,
    "QTime":0},
  "response":{
    "znodeVersion":0,
    "params":{"x":{
        "a":"A val",
        "b":"B val",
        "":{"v":0}}}}}
	at __randomizedtesting.SeedInfo.seed([834D68B5BA8DEE36:B19576F147183CE]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:399)
	at org.apache.solr.handler.TestSolrConfigHandlerCloud.testReqParams(TestSolrConfigHandlerCloud.java:224)
	at org.apache.solr.handler.TestSolrConfigHandlerCloud.test(TestSolrConfigHandlerCloud.java:78)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:958)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:933)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10730 lines...]
   [junit4] Suite: org.apache.solr.handler.TestSolrConfigHandlerCloud
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/init-core-data-001
   [junit4]   2> 1218369 T6428 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1218369 T6428 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1218373 T6428 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1218373 T6429 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1218473 T6428 oasc.ZkTestServer.run start zk server on port:58069
   [junit4]   2> 1218474 T6428 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1218475 T6428 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1218476 T6436 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@152b21e4 name:ZooKeeperConnection Watcher:127.0.0.1:58069 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1218476 T6428 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1218477 T6428 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1218477 T6428 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1218478 T6428 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1218479 T6428 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1218479 T6439 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e08e631 name:ZooKeeperConnection Watcher:127.0.0.1:58069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1218480 T6428 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1218480 T6428 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1218480 T6428 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1218481 T6428 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1218481 T6428 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1218482 T6428 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1218482 T6428 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1218483 T6428 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1218483 T6428 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1218484 T6428 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1218484 T6428 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1218485 T6428 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1218485 T6428 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1218485 T6428 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1218486 T6428 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1218486 T6428 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1218487 T6428 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1218487 T6428 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1218488 T6428 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1218488 T6428 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1218488 T6428 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1218489 T6428 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1218489 T6428 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1218489 T6428 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1218490 T6428 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1218490 T6428 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1218491 T6428 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1218491 T6428 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1218532 T6428 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/cores/collection1
   [junit4]   2> 1218533 T6428 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1218535 T6428 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36080
   [junit4]   2> 1218536 T6428 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/tempDir-001/control/data, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/cores, hostContext=/, hostPort=36080}
   [junit4]   2> 1218536 T6428 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@2792e317
   [junit4]   2> 1218536 T6428 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/'
   [junit4]   2> 1218548 T6428 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/solr.xml
   [junit4]   2> 1218555 T6428 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/cores
   [junit4]   2> 1218555 T6428 oasc.CoreContainer.<init> New CoreContainer 199768094
   [junit4]   2> 1218555 T6428 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/]
   [junit4]   2> 1218555 T6428 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/lib
   [junit4]   2> 1218555 T6428 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/lib).
   [junit4]   2> 1218560 T6428 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1218560 T6428 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1218560 T6428 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1218560 T6428 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1218560 T6428 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1218560 T6428 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1218560 T6428 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1218560 T6428 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1218561 T6428 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1218561 T6428 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1218561 T6428 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 1218561 T6428 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1218561 T6428 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1218561 T6428 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1218562 T6428 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 1218562 T6428 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58069/solr
   [junit4]   2> 1218562 T6428 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1218562 T6428 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1218562 T6428 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1218564 T6451 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@712aa275 name:ZooKeeperConnection Watcher:127.0.0.1:58069 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1218564 T6428 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1218564 T6428 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1218565 T6428 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1218565 T6454 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3747a9e5 name:ZooKeeperConnection Watcher:127.0.0.1:58069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1218566 T6428 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1218566 T6428 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1218567 T6428 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1218568 T6428 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1218569 T6428 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1218569 T6428 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1218571 T6428 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1218572 T6428 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1218572 T6428 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1218573 T6428 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36080_
   [junit4]   2> 1218573 T6428 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36080_
   [junit4]   2> 1218574 T6428 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1218574 T6428 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1218575 T6428 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1218575 T6428 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:36080_
   [junit4]   2> 1218575 T6428 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1218576 T6428 oasc.Overseer.start Overseer (id=93532736942178307-127.0.0.1:36080_-n_0000000000) starting
   [junit4]   2> 1218576 T6428 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1218579 T6428 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1218579 T6428 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1218579 T6456 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1218579 T6455 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1218580 T6428 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/cores
   [junit4]   2> 1218581 T6428 oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {coreNodeName=, loadOnStartup=true, schema=schema.xml, collection=control_collection, instanceDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/cores/collection1, name=collection1, transient=false, config=solrconfig.xml, dataDir=data/, shard=, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/cores/collection1/}
   [junit4]   2> 1218581 T6428 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/cores/collection1/
   [junit4]   2> 1218581 T6428 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1218582 T6458 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 1218582 T6458 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1218583 T6458 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1218583 T6454 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1218583 T6455 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36080_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36080"} current state version: 0
   [junit4]   2> 1218584 T6455 oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36080_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36080"}
   [junit4]   2> 1218584 T6455 oasco.ClusterStateMutator.createCollection building a new cName: control_collection
   [junit4]   2> 1218584 T6455 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1218585 T6454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1219583 T6458 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1219583 T6458 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1219584 T6458 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1219584 T6458 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1219584 T6458 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1219584 T6458 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/cores/collection1/'
   [junit4]   2> 1219593 T6458 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1219595 T6458 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1219598 T6458 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 1219605 T6458 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.1.0
   [junit4]   2> 1219619 T6458 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1219620 T6458 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1219624 T6458 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1219762 T6458 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1219766 T6458 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1219767 T6458 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1219771 T6458 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1219790 T6458 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1219791 T6458 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1219792 T6458 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1219792 T6458 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1219792 T6458 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1219792 T6458 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1219792 T6458 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1219793 T6458 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1219793 T6458 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1219793 T6458 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/cores/collection1/, dataDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/cores/collection1/data/
   [junit4]   2> 1219793 T6458 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@26b994ff
   [junit4]   2> 1219793 T6458 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/cores/collection1/data
   [junit4]   2> 1219794 T6458 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1219794 T6458 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1219794 T6458 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/control-001/cores/collection1/data/index
   [junit4]   2> 1219794 T6458 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=37.6533203125, floorSegmentMB=0.40234375, forceMergeDeletesPctAllowed=29.052814130247018, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1219795 T6458 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@736bbe9b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@26aeed50),segFN=segments_1,generation=1}
   [junit4]   2> 1219795 T6458 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1219798 T6458 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1219798 T6458 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1219798 T6458 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1219798 T6458 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1219798 T6458 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1219798 T6458 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1219799 T6458 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1219799 T6458 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1219799 T6458 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1219801 T6458 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1219801 T6458 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1219802 T6458 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1219803 T6458 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1219805 T6458 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /get,/update/csv,/admin/segments,/admin/mbeans,/admin/system,/admin/luke,/update/json,/admin/plugins,/schema,/admin/properties,/update/json/docs,/admin/threads,/config,/replication,/update,/admin/logging,/admin/ping,standard,/admin/file
   [junit4]   2> 1219806 T6458 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1219806 T6458 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1219806 T6458 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
   [junit4]   2> 1219807 T6458 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1219807 T6458 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1219807 T6458 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20933398552244942]
   [junit4]   2> 1219807 T6458 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@736bbe9b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@26aeed50),segFN=segments_1,generation=1}
   [junit4]   2> 1219807 T6458 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1219808 T6458 oass.SolrIndexSearcher.<init> Opening Searcher@3265c76c[collection1] main
   [junit4]   2> 1219808 T6458 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1219808 T6458 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1219808 T6458 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1219808 T6458 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1219808 T6458 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1219809 T6458 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1219809 T6458 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1219809 T6458 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1219809 T6458 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1219809 T6458 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1219810 T6459 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3265c76c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1219810 T6458 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1219810 T6462 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36080 collection:control_collection shard:shard1
   [junit4]   2> 1219810 T6428 oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1219811 T6428 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1219811 T6462 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1219811 T6428 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1219811 T6428 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1219812 T6465 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5448d254 name:ZooKeeperConnection Watcher:127.0.0.1:58069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1219812 T6428 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1219813 T6428 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1219813 T6428 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1219813 T6462 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1219814 T6454 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1219814 T6462 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1219815 T6462 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1219815 T6455 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"} current state version: 1
   [junit4]   2> ASYNC  NEW_CORE C3031 name=collection1 org.apache.solr.core.SolrCore@5424e456 url=http://127.0.0.1:36080/collection1 node=127.0.0.1:36080_ C3031_STATE=coll:control_collection core:collection1 props:{state=down, node_name=127.0.0.1:36080_, core=collection1, base_url=http://127.0.0.1:36080}
   [junit4]   2> 1219815 T6462 C3031 P36080 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:36080/collection1/
   [junit4]   2> 1219815 T6428 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1219815 T6462 C3031 P36080 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1219815 T6462 C3031 P36080 oasc.SyncStrategy.syncToMe http://127.0.0.1:36080/collection1/ has no replicas
   [junit4]   2> 1219815 T6462 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:36080/collection1/ shard1
   [junit4]   2> 1219815 T6462 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1219817 T6454 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1219817 T6455 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36080",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 1
   [junit4]   2> 1219856 T6428 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/cores/collection1
   [junit4]   2> 1219857 T6428 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001
   [junit4]   2> 1219857 T6428 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1219858 T6428 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48607
   [junit4]   2> 1219858 T6428 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/tempDir-001/jetty1, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/cores, hostContext=/, solrconfig=solrconfig.xml, hostPort=48607}
   [junit4]   2> 1219858 T6428 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@2792e317
   [junit4]   2> 1219859 T6428 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/'
   [junit4]   2> 1219870 T6428 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/solr.xml
   [junit4]   2> 1219877 T6428 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/cores
   [junit4]   2> 1219877 T6428 oasc.CoreContainer.<init> New CoreContainer 1907489859
   [junit4]   2> 1219877 T6428 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/]
   [junit4]   2> 1219878 T6428 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/lib
   [junit4]   2> 1219878 T6428 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/lib).
   [junit4]   2> 1219882 T6428 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1219882 T6428 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1219882 T6428 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1219883 T6428 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1219883 T6428 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1219883 T6428 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1219883 T6428 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1219883 T6428 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1219883 T6428 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1219883 T6428 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1219883 T6428 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 1219884 T6428 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1219884 T6428 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1219884 T6428 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1219884 T6428 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 1219884 T6428 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58069/solr
   [junit4]   2> 1219885 T6428 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1219885 T6428 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1219885 T6428 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1219886 T6477 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2571c019 name:ZooKeeperConnection Watcher:127.0.0.1:58069 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1219886 T6428 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1219887 T6428 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1219887 T6428 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1219888 T6480 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3be93cd3 name:ZooKeeperConnection Watcher:127.0.0.1:58069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1219888 T6428 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1219890 T6428 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1219919 T6480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1219919 T6454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1219919 T6465 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1219967 T6462 oasc.ZkController.register We are http://127.0.0.1:36080/collection1/ and leader is http://127.0.0.1:36080/collection1/
   [junit4]   2> 1219967 T6462 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36080
   [junit4]   2> 1219968 T6462 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1219968 T6462 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 1219968 T6462 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1219968 T6454 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1219969 T6455 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:36080_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36080"} current state version: 2
   [junit4]   2> 1219969 T6455 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:36080_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36080"}
   [junit4]   2> 1220070 T6454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1220070 T6480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1220070 T6465 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1220892 T6428 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48607_
   [junit4]   2> 1220892 T6428 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48607_
   [junit4]   2> 1220893 T6428 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1220895 T6428 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/cores
   [junit4]   2> 1220895 T6428 oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {dataDir=data/, transient=false, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/cores/collection1/, collection=collection1, instanceDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/cores/collection1, shard=, loadOnStartup=true, coreNodeName=, schema=schema.xml, config=solrconfig.xml, name=collection1}
   [junit4]   2> 1220896 T6428 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/cores/collection1/
   [junit4]   2> 1220896 T6428 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1220896 T6481 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1220896 T6481 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1220897 T6454 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1220897 T6481 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1220897 T6455 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48607_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48607"} current state version: 3
   [junit4]   2> 1220897 T6455 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:48607_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48607"}
   [junit4]   2> 1220898 T6455 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 1220898 T6455 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1220999 T6465 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1220999 T6454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1220999 T6480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1221897 T6481 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1221897 T6481 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1221898 T6481 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1221898 T6481 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1221898 T6481 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1221899 T6481 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/cores/collection1/'
   [junit4]   2> 1221910 T6481 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1221912 T6481 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1221916 T6481 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 1221928 T6481 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.1.0
   [junit4]   2> 1221952 T6481 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1221953 T6481 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1221958 T6481 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1222135 T6481 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1222139 T6481 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1222140 T6481 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1222144 T6481 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1222146 T6481 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1222147 T6481 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1222148 T6481 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1222148 T6481 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1222148 T6481 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1222148 T6481 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1222148 T6481 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1222149 T6481 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1222149 T6481 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1222149 T6481 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/cores/collection1/, dataDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/cores/collection1/data/
   [junit4]   2> 1222149 T6481 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@26b994ff
   [junit4]   2> 1222150 T6481 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/cores/collection1/data
   [junit4]   2> 1222150 T6481 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/cores/collection1/data/index/
   [junit4]   2> 1222150 T6481 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1222150 T6481 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-1-001/cores/collection1/data/index
   [junit4]   2> 1222151 T6481 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=37.6533203125, floorSegmentMB=0.40234375, forceMergeDeletesPctAllowed=29.052814130247018, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1222151 T6481 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@55d6e26 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@38ea838a),segFN=segments_1,generation=1}
   [junit4]   2> 1222151 T6481 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1222155 T6481 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1222155 T6481 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1222155 T6481 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1222156 T6481 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1222156 T6481 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1222156 T6481 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1222156 T6481 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1222156 T6481 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1222156 T6481 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1222158 T6481 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1222159 T6481 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1222160 T6481 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1222161 T6481 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1222164 T6481 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /get,/update/csv,/admin/segments,/admin/mbeans,/admin/system,/admin/luke,/update/json,/admin/plugins,/schema,/admin/properties,/update/json/docs,/admin/threads,/config,/replication,/update,/admin/logging,/admin/ping,standard,/admin/file
   [junit4]   2> 1222165 T6481 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1222166 T6481 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1222166 T6481 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
   [junit4]   2> 1222166 T6481 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1222166 T6481 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1222166 T6481 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20933398552244942]
   [junit4]   2> 1222167 T6481 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@55d6e26 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@38ea838a),segFN=segments_1,generation=1}
   [junit4]   2> 1222167 T6481 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1222167 T6481 oass.SolrIndexSearcher.<init> Opening Searcher@3080c813[collection1] main
   [junit4]   2> 1222167 T6481 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1222168 T6481 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1222168 T6481 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1222168 T6481 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1222168 T6481 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1222168 T6481 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1222168 T6481 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1222169 T6481 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1222169 T6481 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1222169 T6481 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1222169 T6482 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3080c813[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1222169 T6481 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1222170 T6485 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48607 collection:collection1 shard:shard1
   [junit4]   2> 1222170 T6428 oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1222170 T6428 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1222170 T6485 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1222172 T6485 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1222173 T6454 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1222173 T6485 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1222173 T6485 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3032 name=collection1 org.apache.solr.core.SolrCore@7a6822e1 url=http://127.0.0.1:48607/collection1 node=127.0.0.1:48607_ C3032_STATE=coll:collection1 core:collection1 props:{state=down, node_name=127.0.0.1:48607_, core=collection1, base_url=http://127.0.0.1:48607}
   [junit4]   2> 1222173 T6485 C3032 P48607 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48607/collection1/
   [junit4]   2> 1222173 T6455 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1"} current state version: 4
   [junit4]   2> 1222173 T6485 C3032 P48607 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1222173 T6485 C3032 P48607 oasc.SyncStrategy.syncToMe http://127.0.0.1:48607/collection1/ has no replicas
   [junit4]   2> 1222173 T6485 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48607/collection1/ shard1
   [junit4]   2> 1222174 T6485 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1222175 T6454 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1222175 T6455 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48607",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 4
   [junit4]   2> 1222213 T6428 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/cores/collection1
   [junit4]   2> 1222213 T6428 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2 in directory /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001
   [junit4]   2> 1222214 T6428 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1222215 T6428 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38500
   [junit4]   2> 1222215 T6428 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {hostContext=/, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/cores, solrconfig=solrconfig.xml, hostPort=38500, solr.data.dir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/tempDir-001/jetty2}
   [junit4]   2> 1222215 T6428 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@2792e317
   [junit4]   2> 1222216 T6428 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/'
   [junit4]   2> 1222227 T6428 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/solr.xml
   [junit4]   2> 1222234 T6428 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/cores
   [junit4]   2> 1222234 T6428 oasc.CoreContainer.<init> New CoreContainer 1477867605
   [junit4]   2> 1222234 T6428 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/]
   [junit4]   2> 1222234 T6428 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/lib
   [junit4]   2> 1222235 T6428 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/lib).
   [junit4]   2> 1222239 T6428 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1222239 T6428 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1222239 T6428 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1222239 T6428 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1222239 T6428 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1222240 T6428 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1222240 T6428 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1222240 T6428 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1222240 T6428 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1222240 T6428 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1222240 T6428 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 1222240 T6428 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1222241 T6428 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1222241 T6428 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1222241 T6428 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 1222241 T6428 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58069/solr
   [junit4]   2> 1222241 T6428 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1222241 T6428 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1222242 T6428 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1222243 T6497 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@578ecc84 name:ZooKeeperConnection Watcher:127.0.0.1:58069 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1222243 T6428 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1222243 T6428 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1222244 T6428 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1222245 T6500 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@292f0bfe name:ZooKeeperConnection Watcher:127.0.0.1:58069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1222245 T6428 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1222247 T6428 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1222277 T6454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1222277 T6500 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1222277 T6480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1222277 T6465 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1222326 T6485 oasc.ZkController.register We are http://127.0.0.1:48607/collection1/ and leader is http://127.0.0.1:48607/collection1/
   [junit4]   2> 1222326 T6485 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48607
   [junit4]   2> 1222326 T6485 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1222326 T6485 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1222326 T6485 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1222327 T6454 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1222327 T6455 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:48607_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48607"} current state version: 5
   [junit4]   2> 1222327 T6455 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:48607_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48607"}
   [junit4]   2> 1222429 T6454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1222429 T6500 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1222429 T6480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1222429 T6465 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1223248 T6428 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38500_
   [junit4]   2> 1223249 T6428 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38500_
   [junit4]   2> 1223250 T6428 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1223251 T6428 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/cores
   [junit4]   2> 1223251 T6428 oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {config=solrconfig.xml, instanceDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/cores/collection1, collection=collection1, dataDir=data/, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/cores/collection1/, transient=false, schema=schema.xml, name=collection1, shard=, loadOnStartup=true, coreNodeName=}
   [junit4]   2> 1223252 T6428 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/cores/collection1/
   [junit4]   2> 1223252 T6428 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1223252 T6501 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1223252 T6501 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1223253 T6501 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1223253 T6454 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1223253 T6455 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38500_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38500"} current state version: 6
   [junit4]   2> 1223254 T6455 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38500_",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38500"}
   [junit4]   2> 1223254 T6455 oasco.ReplicaMutator.updateState Collection already exists with numShards=2
   [junit4]   2> 1223254 T6455 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1223355 T6500 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1223355 T6465 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1223355 T6454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1223355 T6480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1224253 T6501 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1224253 T6501 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1224253 T6501 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1224254 T6501 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1224255 T6501 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1224255 T6501 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/cores/collection1/'
   [junit4]   2> 1224264 T6501 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1224265 T6501 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1224268 T6501 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 1224310 T6501 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.1.0
   [junit4]   2> 1224326 T6501 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1224327 T6501 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1224331 T6501 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1224543 T6501 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1224549 T6501 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1224552 T6501 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1224557 T6501 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1224560 T6501 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1224562 T6501 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1224562 T6501 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1224562 T6501 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1224562 T6501 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1224563 T6501 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1224563 T6501 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1224563 T6501 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1224563 T6501 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1224564 T6501 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/cores/collection1/, dataDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/cores/collection1/data/
   [junit4]   2> 1224564 T6501 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@26b994ff
   [junit4]   2> 1224565 T6501 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/cores/collection1/data
   [junit4]   2> 1224565 T6501 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/cores/collection1/data/index/
   [junit4]   2> 1224565 T6501 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1224566 T6501 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-2-001/cores/collection1/data/index
   [junit4]   2> 1224566 T6501 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=37.6533203125, floorSegmentMB=0.40234375, forceMergeDeletesPctAllowed=29.052814130247018, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1224567 T6501 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2a49939 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4476e8a),segFN=segments_1,generation=1}
   [junit4]   2> 1224567 T6501 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1224572 T6501 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1224572 T6501 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1224573 T6501 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1224573 T6501 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1224573 T6501 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1224573 T6501 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1224574 T6501 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1224574 T6501 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1224574 T6501 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1224577 T6501 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1224578 T6501 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1224579 T6501 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1224580 T6501 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1224586 T6501 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /get,/update/csv,/admin/segments,/admin/mbeans,/admin/system,/admin/luke,/update/json,/admin/plugins,/schema,/admin/properties,/update/json/docs,/admin/threads,/config,/replication,/update,/admin/logging,/admin/ping,standard,/admin/file
   [junit4]   2> 1224587 T6501 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1224587 T6501 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1224588 T6501 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
   [junit4]   2> 1224588 T6501 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1224588 T6501 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1224589 T6501 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20933398552244942]
   [junit4]   2> 1224589 T6501 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2a49939 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4476e8a),segFN=segments_1,generation=1}
   [junit4]   2> 1224589 T6501 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1224590 T6501 oass.SolrIndexSearcher.<init> Opening Searcher@137ec165[collection1] main
   [junit4]   2> 1224590 T6501 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1224590 T6501 oascc.ZkStateReader.readCon

[...truncated too long message...]

with IndexWriterCloser
   [junit4]   2> 1267168 T6428 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1267169 T6428 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1267170 T6428 oasc.CachingDirectoryFactory.closeCacheValue looking to close /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-4-001/cores/collection1/data/index [CachedDir<<refCount=0;path=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-4-001/cores/collection1/data/index;done=false>>]
   [junit4]   2> 1267170 T6428 oasc.CachingDirectoryFactory.close Closing directory: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-4-001/cores/collection1/data/index
   [junit4]   2> 1267171 T6428 oasc.CachingDirectoryFactory.closeCacheValue looking to close /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-4-001/cores/collection1/data [CachedDir<<refCount=0;path=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-4-001/cores/collection1/data;done=false>>]
   [junit4]   2> 1267171 T6428 oasc.CachingDirectoryFactory.close Closing directory: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001/shard-4-001/cores/collection1/data
   [junit4]   2> 1267171 T6428 oasc.Overseer.close Overseer (id=93532736942178316-127.0.0.1:35421_-n_0000000004) closing
   [junit4]   2> 1267172 T6759 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:35421_
   [junit4]   2> 1268672 T6753 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1268704 T6428 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 1268894 T6428 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58069 58069
   [junit4]   2> 1268912 T6429 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58069 58069
   [junit4]   2> 1268913 T6429 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		105	/solr/configs/conf1
   [junit4]   2> 		6	/solr/clusterstate.json
   [junit4]   2> 		6	/solr/aliases.json
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		6	/solr/live_nodes
   [junit4]   2> 		5	/solr/overseer/collection-queue-work
   [junit4]   2> 		5	/solr/overseer/queue
   [junit4]   2> 	
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSolrConfigHandlerCloud -Dtests.method=test -Dtests.seed=834D68B5BA8DEE36 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_ZA -Dtests.timezone=Atlantic/South_Georgia -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 50.5s J0 | TestSolrConfigHandlerCloud.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Could not get expected value  'CY val modified' for path 'response/params/y/c' full output: {
   [junit4]    >   "responseHeader":{
   [junit4]    >     "status":0,
   [junit4]    >     "QTime":0},
   [junit4]    >   "response":{
   [junit4]    >     "znodeVersion":0,
   [junit4]    >     "params":{"x":{
   [junit4]    >         "a":"A val",
   [junit4]    >         "b":"B val",
   [junit4]    >         "":{"v":0}}}}}
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([834D68B5BA8DEE36:B19576F147183CE]:0)
   [junit4]    > 	at org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:399)
   [junit4]    > 	at org.apache.solr.handler.TestSolrConfigHandlerCloud.testReqParams(TestSolrConfigHandlerCloud.java:224)
   [junit4]    > 	at org.apache.solr.handler.TestSolrConfigHandlerCloud.test(TestSolrConfigHandlerCloud.java:78)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:958)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:933)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1268920 T6428 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.handler.TestSolrConfigHandlerCloud 834D68B5BA8DEE36-001
   [junit4]   2> 50553 T6427 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=15657, maxDocsPerChunk=9, blockSize=679), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=15657, blockSize=679)), sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=en_ZA, timezone=Atlantic/South_Georgia
   [junit4]   2> NOTE: Linux 3.13.0-46-generic amd64/Oracle Corporation 1.7.0_76 (64-bit)/cpus=12,threads=1,free=186744176,total=507772928
   [junit4]   2> NOTE: All tests run in this JVM: [BadCopyFieldTest, TestStressLucene, SimplePostToolTest, TestIndexingPerformance, OverriddenZkACLAndCredentialsProvidersTest, ChaosMonkeyNothingIsSafeTest, PeerSyncTest, AtomicUpdatesTest, TestFaceting, TestSolrQueryParserDefaultOperatorResource, ClusterStateTest, URLClassifyProcessorTest, HdfsUnloadDistributedZkTest, TestRandomMergePolicy, TestStandardQParsers, TestUpdate, MBeansHandlerTest, QueryElevationComponentTest, TestExactSharedStatsCache, TestJmxMonitoredMap, UUIDFieldTest, SolrIndexConfigTest, TestRandomFaceting, TestSolrDeletionPolicy1, TestConfig, ParsingFieldUpdateProcessorsTest, TestShortCircuitedRequests, TermsComponentTest, DateMathParserTest, SignatureUpdateProcessorFactoryTest, TestNRTOpen, ExternalFileFieldSortTest, TestSolr4Spatial, TestSimpleTrackingShardHandler, TestJmxIntegration, TestMissingGroups, TestDistribDocBasedVersion, HdfsBasicDistributedZk2Test, CurrencyFieldXmlFileTest, TestQuerySenderListener, TestDefaultSearchFieldResource, DeleteLastCustomShardedReplicaTest, HighlighterTest, CoreAdminCreateDiscoverTest, UpdateRequestProcessorFactoryTest, TestNamedUpdateProcessors, DistributedFacetPivotSmallAdvancedTest, TestManagedSchemaFieldTypeResource, TestReplicationHandlerBackup, TestWordDelimiterFilterFactory, TestExactStatsCache, DocExpirationUpdateProcessorFactoryTest, DebugComponentTest, IndexBasedSpellCheckerTest, TestCloudSchemaless, TestReversedWildcardFilterFactory, DeleteInactiveReplicaTest, TestMergePolicyConfig, ChangedSchemaMergeTest, TestLazyCores, MultiThreadedOCPTest, CSVRequestHandlerTest, TestRealTimeGet, TestIBSimilarityFactory, MultiTermTest, TestDocumentBuilder, HdfsRecoveryZkTest, MinimalSchemaTest, TestJoin, TestBulkSchemaAPI, ShardSplitTest, ZkNodePropsTest, TestManagedSchemaFieldResource, SuggesterFSTTest, SuggesterWFSTTest, OpenCloseCoreStressTest, OverseerTest, FullSolrCloudDistribCmdsTest, TestDistributedSearch, TestCoreDiscovery, CoreAdminHandlerTest, SuggesterTSTTest, TestTrie, TestCSVLoader, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, DocValuesMultiTest, SolrInfoMBeanTest, DocumentAnalysisRequestHandlerTest, DocumentBuilderTest, FieldAnalysisRequestHandlerTest, FastVectorHighlighterTest, LoggingHandlerTest, ReturnFieldsTest, BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, SOLR749Test, TestSolrIndexConfig, SampleTest, TestSearchPerf, TestElisionMultitermQuery, TestPhraseSuggestions, SynonymTokenizerTest, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestDefaultSimilarityFactory, ResourceLoaderTest, PrimUtilsTest, SliceStateTest, TestRTGBase, CursorPagingTest, CollectionsAPIAsyncDistributedZkTest, DistribCursorPagingTest, DistribJoinFromCollectionTest, HttpPartitionTest, TestCloudInspectUtil, TestLeaderElectionZkExpiry, TestMiniSolrCloudClusterSSL, TestRebalanceLeaders, TestReqParamsAPI, TestSolrConfigHandlerCloud]
   [junit4] Completed on J0 in 50.82s, 1 test, 1 failure <<< FAILURES!

[...truncated 150 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:529: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:477: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:61: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build.xml:191: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/common-build.xml:509: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:1370: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:976: There were test failures: 475 suites, 1892 tests, 1 failure, 48 ignored (21 assumptions)

Total time: 41 minutes 5 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.7.0_76 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.8.0_60-ea-b06) - Build # 11905 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11905/
Java: 32bit/jdk1.8.0_60-ea-b06 -server -XX:+UseParallelGC

2 tests failed.
FAILED:  org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test

Error Message:
Didn't see all replicas for shard shard1 in collection1 come up within 30000 ms! ClusterState: {   "collection1":{     "replicationFactor":"1",     "shards":{"shard1":{         "range":"80000000-7fffffff",         "state":"active",         "replicas":{           "core_node1":{             "core":"collection1",             "base_url":"http://127.0.0.1:44277/gm_",             "node_name":"127.0.0.1:44277_gm_",             "state":"active",             "leader":"true"},           "core_node2":{             "core":"collection1",             "base_url":"http://127.0.0.1:50263/gm_",             "node_name":"127.0.0.1:50263_gm_",             "state":"recovering"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "autoCreated":"true"},   "control_collection":{     "replicationFactor":"1",     "shards":{"shard1":{         "range":"80000000-7fffffff",         "state":"active",         "replicas":{"core_node1":{             "core":"collection1",             "base_url":"http://127.0.0.1:47753/gm_",             "node_name":"127.0.0.1:47753_gm_",             "state":"active",             "leader":"true"}}}},     "router":{"name":"compositeId"},     "maxShardsPerNode":"1",     "autoAddReplicas":"false",     "autoCreated":"true"}}

Stack Trace:
java.lang.AssertionError: Didn't see all replicas for shard shard1 in collection1 come up within 30000 ms! ClusterState: {
  "collection1":{
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{
          "core_node1":{
            "core":"collection1",
            "base_url":"http://127.0.0.1:44277/gm_",
            "node_name":"127.0.0.1:44277_gm_",
            "state":"active",
            "leader":"true"},
          "core_node2":{
            "core":"collection1",
            "base_url":"http://127.0.0.1:50263/gm_",
            "node_name":"127.0.0.1:50263_gm_",
            "state":"recovering"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "autoCreated":"true"},
  "control_collection":{
    "replicationFactor":"1",
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{"core_node1":{
            "core":"collection1",
            "base_url":"http://127.0.0.1:47753/gm_",
            "node_name":"127.0.0.1:47753_gm_",
            "state":"active",
            "leader":"true"}}}},
    "router":{"name":"compositeId"},
    "maxShardsPerNode":"1",
    "autoAddReplicas":"false",
    "autoCreated":"true"}}
	at __randomizedtesting.SeedInfo.seed([37A2B35D382C6FE4:BFF68C8796D0021C]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.ensureAllReplicasAreActive(AbstractFullDistribZkTestBase.java:1995)
	at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:103)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:958)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:933)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Error from server at http://127.0.0.1:55334/compositeid_collection_with_routerfield_shard1_replica1: no servers hosting shard: 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:55334/compositeid_collection_with_routerfield_shard1_replica1: no servers hosting shard: 
	at __randomizedtesting.SeedInfo.seed([37A2B35D382C6FE4:BFF68C8796D0021C]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:584)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:236)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:228)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:943)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:958)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDeleteByIdCompositeRouterWithRouterField(FullSolrCloudDistribCmdsTest.java:357)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:146)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:958)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:933)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 9957 lines...]
   [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/init-core-data-001
   [junit4]   2> 576177 T1396 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 576179 T1396 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 576180 T1397 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 576280 T1396 oasc.ZkTestServer.run start zk server on port:52056
   [junit4]   2> 576280 T1396 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 576281 T1396 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 576282 T1404 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1341c35 name:ZooKeeperConnection Watcher:127.0.0.1:52056 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 576282 T1396 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 576283 T1396 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 576283 T1396 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 576284 T1396 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 576285 T1396 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 576285 T1407 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c20f20 name:ZooKeeperConnection Watcher:127.0.0.1:52056/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 576285 T1396 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 576286 T1396 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 576286 T1396 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 576287 T1396 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 576287 T1396 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 576288 T1396 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 576288 T1396 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 576288 T1396 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 576289 T1396 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 576290 T1396 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 576290 T1396 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 576291 T1396 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 576291 T1396 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 576291 T1396 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 576292 T1396 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 576292 T1396 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 576293 T1396 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 576293 T1396 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 576294 T1396 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 576294 T1396 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 576294 T1396 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 576295 T1396 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 576295 T1396 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 576295 T1396 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 576296 T1396 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 576296 T1396 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 576297 T1396 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 576297 T1396 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 576342 T1396 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/cores/collection1
   [junit4]   2> 576343 T1396 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 576346 T1396 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48266
   [junit4]   2> 576346 T1396 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/tempDir-001/control/data, hostContext=/, hostPort=48266, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/cores}
   [junit4]   2> 576346 T1396 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@e2f2a
   [junit4]   2> 576346 T1396 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/'
   [junit4]   2> 576355 T1396 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/solr.xml
   [junit4]   2> 576362 T1396 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/cores
   [junit4]   2> 576362 T1396 oasc.CoreContainer.<init> New CoreContainer 9401874
   [junit4]   2> 576363 T1396 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/]
   [junit4]   2> 576363 T1396 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/lib
   [junit4]   2> 576363 T1396 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/lib).
   [junit4]   2> 576368 T1396 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 576368 T1396 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 576368 T1396 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 576368 T1396 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 576368 T1396 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 576368 T1396 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 576369 T1396 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 576369 T1396 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 576369 T1396 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 576369 T1396 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 576369 T1396 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 576370 T1396 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 576370 T1396 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 576370 T1396 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 576370 T1396 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 576371 T1396 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52056/solr
   [junit4]   2> 576371 T1396 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 576371 T1396 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 576371 T1396 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 576372 T1419 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11a6613 name:ZooKeeperConnection Watcher:127.0.0.1:52056 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 576373 T1396 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 576373 T1396 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 576374 T1396 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 576375 T1422 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d833bb name:ZooKeeperConnection Watcher:127.0.0.1:52056/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 576375 T1396 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 576376 T1396 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 576377 T1396 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 576378 T1396 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 576378 T1396 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 576379 T1396 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 576380 T1396 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 576381 T1396 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 576381 T1396 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 576382 T1396 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48266_
   [junit4]   2> 576382 T1396 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48266_
   [junit4]   2> 576383 T1396 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 576383 T1396 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 576384 T1396 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 576385 T1396 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:48266_
   [junit4]   2> 576385 T1396 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 576385 T1396 oasc.Overseer.start Overseer (id=93533441052180483-127.0.0.1:48266_-n_0000000000) starting
   [junit4]   2> 576386 T1396 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 576388 T1396 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 576388 T1424 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 576389 T1423 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 576388 T1396 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 576390 T1396 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/cores
   [junit4]   2> 576391 T1396 oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/cores/collection1, collection=control_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 576391 T1396 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/cores/collection1/
   [junit4]   2> 576391 T1396 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 576391 T1426 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 576392 T1426 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 576392 T1422 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 576392 T1426 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 576392 T1423 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48266",
   [junit4]   2> 	  "node_name":"127.0.0.1:48266_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 0
   [junit4]   2> 576393 T1423 oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48266",
   [junit4]   2> 	  "node_name":"127.0.0.1:48266_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 576393 T1423 oasco.ClusterStateMutator.createCollection building a new cName: control_collection
   [junit4]   2> 576393 T1423 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 576393 T1422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 577392 T1426 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 577392 T1426 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 577393 T1426 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 577393 T1426 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 577393 T1426 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 577393 T1426 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/cores/collection1/'
   [junit4]   2> 577401 T1426 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 577402 T1426 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 577405 T1426 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 577407 T1426 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.1.0
   [junit4]   2> 577412 T1426 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 577413 T1426 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 577415 T1426 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 577458 T1426 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 577459 T1426 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 577460 T1426 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 577461 T1426 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 577468 T1426 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 577468 T1426 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 577468 T1426 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/cores/collection1/, dataDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/cores/collection1/data/
   [junit4]   2> 577469 T1426 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1840c54
   [junit4]   2> 577469 T1426 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/cores/collection1/data
   [junit4]   2> 577469 T1426 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/cores/collection1/data/index/
   [junit4]   2> 577469 T1426 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 577469 T1426 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/control-001/cores/collection1/data/index
   [junit4]   2> 577470 T1426 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=36, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=94.037109375, floorSegmentMB=1.6416015625, forceMergeDeletesPctAllowed=1.8174646722897925, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6098921863086849
   [junit4]   2> 577470 T1426 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@17d59a9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3f76fb),segFN=segments_1,generation=1}
   [junit4]   2> 577470 T1426 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 577472 T1426 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 577473 T1426 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 577473 T1426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 577473 T1426 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 577473 T1426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 577473 T1426 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 577473 T1426 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 577473 T1426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 577473 T1426 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 577475 T1426 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 577475 T1426 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 577476 T1426 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 577476 T1426 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 577478 T1426 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 577478 T1426 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 577479 T1426 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 577479 T1426 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
   [junit4]   2> 577479 T1426 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 577480 T1426 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 577480 T1426 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=43, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.17336836215691626]
   [junit4]   2> 577480 T1426 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@17d59a9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3f76fb),segFN=segments_1,generation=1}
   [junit4]   2> 577480 T1426 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 577481 T1426 oass.SolrIndexSearcher.<init> Opening Searcher@149feda[collection1] main
   [junit4]   2> 577481 T1426 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 577481 T1426 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 577481 T1426 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 577481 T1426 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 577482 T1426 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 577482 T1426 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 577482 T1426 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 577482 T1426 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 577482 T1426 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 577482 T1426 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 577483 T1426 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 577483 T1427 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@149feda[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 577483 T1430 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48266 collection:control_collection shard:shard1
   [junit4]   2> 577484 T1396 oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2
   [junit4]   2> 577484 T1396 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 577484 T1430 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 577484 T1396 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 577485 T1396 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 577485 T1433 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1db25b5 name:ZooKeeperConnection Watcher:127.0.0.1:52056/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 577486 T1396 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 577486 T1430 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 577486 T1396 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 577486 T1396 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 577487 T1430 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 577487 T1430 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 577486 T1422 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> ASYNC  NEW_CORE C199 name=collection1 org.apache.solr.core.SolrCore@7b6ab url=http://127.0.0.1:48266/collection1 node=127.0.0.1:48266_ C199_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:48266, node_name=127.0.0.1:48266_, state=down}
   [junit4]   2> 577487 T1430 C199 P48266 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48266/collection1/
   [junit4]   2> 577487 T1423 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"} current state version: 1
   [junit4]   2> 577487 T1430 C199 P48266 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 577487 T1396 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 577488 T1430 C199 P48266 oasc.SyncStrategy.syncToMe http://127.0.0.1:48266/collection1/ has no replicas
   [junit4]   2> 577488 T1396 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 577488 T1430 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48266/collection1/ shard1
   [junit4]   2> 577488 T1396 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 577488 T1430 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 577489 T1396 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 577490 T1436 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f2ac72 name:ZooKeeperConnection Watcher:127.0.0.1:52056/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 577490 T1396 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 577490 T1396 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 577492 T1422 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 577493 T1423 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 2, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48266",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 1
   [junit4]   2> 577494 T1423 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "stateFormat":"2"} current state version: 1
   [junit4]   2> 577494 T1423 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 577500 T1422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 577500 T1433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 577502 T1422 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 577503 T1423 oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/collection1/state.json
   [junit4]   2> 577504 T1422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 577504 T1433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 577543 T1430 oasc.ZkController.register We are http://127.0.0.1:48266/collection1/ and leader is http://127.0.0.1:48266/collection1/
   [junit4]   2> 577543 T1430 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48266
   [junit4]   2> 577543 T1430 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 577544 T1430 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 577544 T1430 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 577544 T1422 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 577545 T1423 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48266",
   [junit4]   2> 	  "node_name":"127.0.0.1:48266_",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 3
   [junit4]   2> 577545 T1423 oasco.ReplicaMutator.updateState Update state numShards=3 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48266",
   [junit4]   2> 	  "node_name":"127.0.0.1:48266_",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 577549 T1396 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/cores/collection1
   [junit4]   2> 577550 T1396 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001
   [junit4]   2> 577550 T1396 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 577552 T1396 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55334
   [junit4]   2> 577553 T1396 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=55334, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/cores}
   [junit4]   2> 577553 T1396 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@e2f2a
   [junit4]   2> 577553 T1396 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/'
   [junit4]   2> 577561 T1396 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/solr.xml
   [junit4]   2> 577565 T1396 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/cores
   [junit4]   2> 577565 T1396 oasc.CoreContainer.<init> New CoreContainer 25341187
   [junit4]   2> 577565 T1396 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/]
   [junit4]   2> 577566 T1396 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/lib
   [junit4]   2> 577566 T1396 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/lib).
   [junit4]   2> 577571 T1396 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 577571 T1396 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 577571 T1396 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 577571 T1396 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 577572 T1396 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 577572 T1396 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 577572 T1396 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 577572 T1396 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 577572 T1396 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 577572 T1396 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 577573 T1396 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 577573 T1396 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 577573 T1396 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 577573 T1396 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 577574 T1396 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 577574 T1396 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52056/solr
   [junit4]   2> 577574 T1396 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 577574 T1396 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 577578 T1396 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 577579 T1448 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f116a2 name:ZooKeeperConnection Watcher:127.0.0.1:52056 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 577580 T1396 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 577580 T1396 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 577581 T1396 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 577582 T1451 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@281219 name:ZooKeeperConnection Watcher:127.0.0.1:52056/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 577582 T1396 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 577584 T1396 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 577647 T1422 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 577647 T1433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 577647 T1451 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 578586 T1396 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55334_
   [junit4]   2> 578586 T1396 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55334_
   [junit4]   2> 578587 T1396 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 578588 T1396 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/cores
   [junit4]   2> 578589 T1396 oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/cores/collection1, collection=collection1, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 578589 T1396 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/cores/collection1/
   [junit4]   2> 578590 T1396 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 578590 T1452 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 578591 T1452 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 578591 T1422 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 578591 T1452 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 578592 T1423 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55334",
   [junit4]   2> 	  "node_name":"127.0.0.1:55334_",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 578592 T1452 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 578592 T1423 oasco.ReplicaMutator.updateState Update state numShards=3 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55334",
   [junit4]   2> 	  "node_name":"127.0.0.1:55334_",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 578592 T1452 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 578592 T1423 oasco.ReplicaMutator.updateState Collection already exists with numShards=3
   [junit4]   2> 578592 T1423 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 578592 T1452 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 
   [junit4]   2> 578592 T1452 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 578693 T1423 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 0
   [junit4]   2> 578694 T1451 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 2)
   [junit4]   2> 578694 T1451 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 
   [junit4]   2> 579593 T1452 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 579593 T1452 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 579594 T1452 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 579594 T1452 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 579594 T1452 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 579594 T1452 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/cores/collection1/'
   [junit4]   2> 579601 T1452 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 579602 T1452 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 579605 T1452 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 579607 T1452 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.1.0
   [junit4]   2> 579612 T1452 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 579613 T1452 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 579616 T1452 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 579658 T1452 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 579658 T1452 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 579659 T1452 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 579660 T1452 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 579669 T1452 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 579670 T1452 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 579670 T1452 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/cores/collection1/, dataDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/cores/collection1/data/
   [junit4]   2> 579670 T1452 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1840c54
   [junit4]   2> 579670 T1452 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/cores/collection1/data
   [junit4]   2> 579671 T1452 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/cores/collection1/data/index/
   [junit4]   2> 579671 T1452 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 579671 T1452 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-1-001/cores/collection1/data/index
   [junit4]   2> 579672 T1452 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=36, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=94.037109375, floorSegmentMB=1.6416015625, forceMergeDeletesPctAllowed=1.8174646722897925, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6098921863086849
   [junit4]   2> 579672 T1452 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1766496 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@11cec1b),segFN=segments_1,generation=1}
   [junit4]   2> 579672 T1452 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 579674 T1452 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 579674 T1452 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 579675 T1452 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 579675 T1452 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 579675 T1452 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 579675 T1452 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 579675 T1452 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 579675 T1452 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 579675 T1452 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 579677 T1452 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 579678 T1452 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 579679 T1452 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 579680 T1452 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 579683 T1452 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 579683 T1452 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 579684 T1452 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 579684 T1452 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
   [junit4]   2> 579684 T1452 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 579684 T1452 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 579685 T1452 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=43, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.17336836215691626]
   [junit4]   2> 579685 T1452 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1766496 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@11cec1b),segFN=segments_1,generation=1}
   [junit4]   2> 579686 T1452 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 579686 T1452 oass.SolrIndexSearcher.<init> Opening Searcher@2d7629[collection1] main
   [junit4]   2> 579686 T1452 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 579686 T1452 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 579686 T1452 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 579687 T1452 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 579687 T1452 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 579687 T1452 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 579687 T1452 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 579687 T1452 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 579688 T1452 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 579688 T1452 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 579688 T1453 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2d7629[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 579688 T1452 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 579689 T1456 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55334 collection:collection1 shard:shard2
   [junit4]   2> 579689 T1396 oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2
   [junit4]   2> 579689 T1396 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 579689 T1456 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 579691 T1456 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 579692 T1422 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 579692 T1456 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 579692 T1456 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C200 name=collection1 org.apache.solr.core.SolrCore@15ad828 url=http://127.0.0.1:55334/collection1 node=127.0.0.1:55334_ C200_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:55334, node_name=127.0.0.1:55334_, state=down}
   [junit4]   2> 579692 T1456 C200 P55334 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55334/collection1/
   [junit4]   2> 579692 T1423 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"} current state version: 4
   [junit4]   2> 579692 T1456 C200 P55334 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 579692 T1456 C200 P55334 oasc.SyncStrategy.syncToMe http://127.0.0.1:55334/collection1/ has no replicas
   [junit4]   2> 579693 T1456 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55334/collection1/ shard2
   [junit4]   2> 579693 T1456 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 579693 T1423 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 1
   [junit4]   2> 579693 T1451 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 2)
   [junit4]   2> 579694 T1451 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2 
   [junit4]   2> 579694 T1423 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55334",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 4
   [junit4]   2> 579695 T1423 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 2
   [junit4]   2> 579695 T1451 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 2)
   [junit4]   2> 579695 T1451 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 579696 T1422 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 579745 T1456 oasc.ZkController.register We are http://127.0.0.1:55334/collection1/ and leader is http://127.0.0.1:55334/collection1/
   [junit4]   2> 579745 T1456 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55334
   [junit4]   2> 579745 T1456 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 579745 T1456 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 579745 T1456 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 579746 T1422 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 579747 T1423 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55334",
   [junit4]   2> 	  "node_name":"127.0.0.1:55334_",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 579747 T1396 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001/cores/collection1
   [junit4]   2> 579747 T1423 oasco.ReplicaMutator.updateState Update state numShards=3 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55334",
   [junit4]   2> 	  "node_name":"127.0.0.1:55334_",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 579747 T1456 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 579748 T1396 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2 in directory /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001
   [junit4]   2> 579748 T1423 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 3
   [junit4]   2> 579748 T1396 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 579748 T1451 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 2)
   [junit4]   2> 579749 T1451 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 579750 T1396 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41008
   [junit4]   2> 579750 T1396 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=41008, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001/cores}
   [junit4]   2> 579751 T1396 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@e2f2a
   [junit4]   2> 579751 T1396 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001/'
   [junit4]   2> 579760 T1396 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001/solr.xml
   [junit4]   2> 579764 T1396 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001/cores
   [junit4]   2> 579764 T1396 oasc.CoreContainer.<init> New CoreContainer 7909030
   [junit4]   2> 579764 T1396 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001/]
   [junit4]   2> 579764 T1396 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001/lib
   [junit4]   2> 579765 T1396 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001/lib).
   [junit4]   2> 579769 T1396 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 579770 T1396 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 579770 T1396 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 579770 T1396 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 579770 T1396 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 579770 T1396 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 579770 T1396 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 579771 T1396 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 579771 T1396 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 579771 T1396 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 579771 T1396 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 579772 T1396 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 579772 T1396 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 579772 T1396 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 579772 T1396 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 579773 T1396 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52056/solr
   [junit4]   2> 579773 T1396 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 579773 T1396 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 579774 T1396 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 579774 T1468 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@130dadc name:ZooKeeperConnection Watcher:127.0.0.1:52056 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 579775 T1396 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 579775 T1396 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 579777 T1396 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 579777 T1471 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@115a9fc name:ZooKeeperConnection Watcher:127.0.0.1:52056/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 579777 T1396 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 579779 T1396 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 580781 T1396 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41008_
   [junit4]   2> 580782 T1396 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41008_
   [junit4]   2> 580783 T1396 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 580784 T1396 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001/cores
   [junit4]   2> 580785 T1396 oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001/cores/collection1, collection=collection1, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 580785 T1396 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001/cores/collection1/
   [junit4]   2> 580785 T1396 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 580786 T1472 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 580786 T1472 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 580787 T1422 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 580787 T1472 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 580787 T1472 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 580787 T1423 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41008",
   [junit4]   2> 	  "node_name":"127.0.0.1:41008_",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 580787 T1472 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 580787 T1423 oasco.ReplicaMutator.updateState Update state numShards=3 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41008",
   [junit4]   2> 	  "node_name":"127.0.0.1:41008_",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 580787 T1423 oasco.ReplicaMutator.updateState Collection already exists with numShards=3
   [junit4]   2> 580788 T1472 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 580788 T1423 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 580788 T1472 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 580788 T1423 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 4
   [junit4]   2> 580788 T1471 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 3)
   [junit4]   2> 580788 T1451 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 3)
   [junit4]   2> 580789 T1471 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 580789 T1451 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 581788 T1472 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 581788 T1472 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 581789 T1472 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 581789 T1472 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 581789 T1472 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 581789 T1472 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001/cores/collection1/'
   [junit4]   2> 581796 T1472 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 581797 T1472 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 581800 T1472 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 581803 T1472 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.1.0
   [junit4]   2> 581809 T1472 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 581810 T1472 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 581812 T1472 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 581861 T1472 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 581861 T1472 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 581862 T1472 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 581863 T1472 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 581869 T1472 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 581869 T1472 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 581870 T1472 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001/cores/collection1/, dataDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001/cores/collection1/data/
   [junit4]   2> 581870 T1472 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1840c54
   [junit4]   2> 581871 T1472 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001/cores/collection1/data
   [junit4]   2> 581871 T1472 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 37A2B35D382C6FE4-001/shard-2-001/cores/collection1/data/index/
   [junit4]   2> 581871 T1472 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/

[...truncated too long message...]

2> 1171606 T5860 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:42554 42554
   [junit4]   2> 1171648 T5861 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:42554 42554
   [junit4]   2> 1171650 T5861 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		4	/solr/aliases.json
   [junit4]   2> 		4	/solr/clusterstate.json
   [junit4]   2> 		3	/solr/configs/conf1
   [junit4]   2> 		2	/solr/collections/collection1/state.json
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		4	/solr/live_nodes
   [junit4]   2> 		3	/solr/overseer/queue
   [junit4]   2> 		3	/solr/overseer/collection-queue-work
   [junit4]   2> 	
   [junit4]   2> 1171650 T5860 oasc.SocketProxy.close WARN Closing 7 connections to: http://127.0.0.1:44277/gm_
   [junit4]   2> 1171650 T5860 oasc.SocketProxy.close WARN Closing 0 connections to: http://127.0.0.1:47753/gm_
   [junit4]   2> 1171651 T5860 oasc.SocketProxy.close WARN Closing 1 connections to: http://127.0.0.1:50263/gm_
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=RecoveryAfterSoftCommitTest -Dtests.method=test -Dtests.seed=37A2B35D382C6FE4 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=be_BY -Dtests.timezone=ECT -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 56.4s J0 | RecoveryAfterSoftCommitTest.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Didn't see all replicas for shard shard1 in collection1 come up within 30000 ms! ClusterState: {
   [junit4]    >   "collection1":{
   [junit4]    >     "replicationFactor":"1",
   [junit4]    >     "shards":{"shard1":{
   [junit4]    >         "range":"80000000-7fffffff",
   [junit4]    >         "state":"active",
   [junit4]    >         "replicas":{
   [junit4]    >           "core_node1":{
   [junit4]    >             "core":"collection1",
   [junit4]    >             "base_url":"http://127.0.0.1:44277/gm_",
   [junit4]    >             "node_name":"127.0.0.1:44277_gm_",
   [junit4]    >             "state":"active",
   [junit4]    >             "leader":"true"},
   [junit4]    >           "core_node2":{
   [junit4]    >             "core":"collection1",
   [junit4]    >             "base_url":"http://127.0.0.1:50263/gm_",
   [junit4]    >             "node_name":"127.0.0.1:50263_gm_",
   [junit4]    >             "state":"recovering"}}}},
   [junit4]    >     "router":{"name":"compositeId"},
   [junit4]    >     "maxShardsPerNode":"1",
   [junit4]    >     "autoAddReplicas":"false",
   [junit4]    >     "autoCreated":"true"},
   [junit4]    >   "control_collection":{
   [junit4]    >     "replicationFactor":"1",
   [junit4]    >     "shards":{"shard1":{
   [junit4]    >         "range":"80000000-7fffffff",
   [junit4]    >         "state":"active",
   [junit4]    >         "replicas":{"core_node1":{
   [junit4]    >             "core":"collection1",
   [junit4]    >             "base_url":"http://127.0.0.1:47753/gm_",
   [junit4]    >             "node_name":"127.0.0.1:47753_gm_",
   [junit4]    >             "state":"active",
   [junit4]    >             "leader":"true"}}}},
   [junit4]    >     "router":{"name":"compositeId"},
   [junit4]    >     "maxShardsPerNode":"1",
   [junit4]    >     "autoAddReplicas":"false",
   [junit4]    >     "autoCreated":"true"}}
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([37A2B35D382C6FE4:BFF68C8796D0021C]:0)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.ensureAllReplicasAreActive(AbstractFullDistribZkTestBase.java:1995)
   [junit4]    > 	at org.apache.solr.cloud.RecoveryAfterSoftCommitTest.test(RecoveryAfterSoftCommitTest.java:103)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:958)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:933)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1171664 T5860 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RecoveryAfterSoftCommitTest 37A2B35D382C6FE4-001
   [junit4]   2> 56422 T5859 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {_version_=Lucene50(blocksize=128), multiDefault=PostingsFormat(name=LuceneVarGapDocFreqInterval), a_t=Lucene50(blocksize=128), intDefault=Lucene50(blocksize=128), id=FSTOrd50, text=PostingsFormat(name=Asserting), range_facet_l=Lucene50(blocksize=128), timestamp=Lucene50(blocksize=128)}, docValues:{timestamp=DocValuesFormat(name=Asserting)}, sim=DefaultSimilarity, locale=be_BY, timezone=ECT
   [junit4]   2> NOTE: Linux 3.13.0-46-generic i386/Oracle Corporation 1.8.0_60-ea (32-bit)/cpus=12,threads=1,free=57279808,total=273154048
   [junit4]   2> NOTE: All tests run in this JVM: [TestExpandComponent, TestNonDefinedSimilarityFactory, TestManagedResourceStorage, TestDFRSimilarityFactory, ParsingFieldUpdateProcessorsTest, MultiTermTest, DistributedFacetPivotSmallTest, DateMathParserTest, TestUpdate, DistributedIntervalFacetingTest, TestSolr4Spatial, TestDistributedSearch, UUIDUpdateProcessorFallbackTest, TestCloudManagedSchema, TestSolrQueryParserResource, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestIndexingPerformance, TestDefaultSearchFieldResource, RequestHandlersTest, HighlighterConfigTest, PeerSyncTest, DeleteShardTest, PrimitiveFieldTypeTest, TestStandardQParsers, ClusterStateTest, EnumFieldTest, TestDefaultStatsCache, TimeZoneUtilsTest, TestCodecSupport, InfoHandlerTest, SolrIndexConfigTest, HdfsBasicDistributedZkTest, PingRequestHandlerTest, SharedFSAutoReplicaFailoverUtilsTest, CSVRequestHandlerTest, TestRandomDVFaceting, StandardRequestHandlerTest, CoreMergeIndexesAdminHandlerTest, HdfsUnloadDistributedZkTest, SyncSliceTest, TestConfigReload, TestBinaryField, AddSchemaFieldsUpdateProcessorFactoryTest, TestCSVLoader, TestRestManager, TestQuerySenderListener, TestLazyCores, BasicDistributedZkTest, TestRemoteStreaming, AnalysisErrorHandlingTest, SolrCoreTest, SuggesterFSTTest, TestInfoStreamLogging, DistributedFacetPivotLargeTest, TestRangeQuery, SignatureUpdateProcessorFactoryTest, TestFileDictionaryLookup, IndexSchemaTest, QueryElevationComponentTest, TestFunctionQuery, TestIntervalFaceting, URLClassifyProcessorTest, TestWordDelimiterFilterFactory, SharedFSAutoReplicaFailoverTest, TestManagedSynonymFilterFactory, SuggesterWFSTTest, TestConfig, TestManagedSchemaFieldTypeResource, CurrencyFieldXmlFileTest, TermVectorComponentTest, RemoteQueryErrorTest, MigrateRouteKeyTest, SuggestComponentTest, TestImplicitCoreProperties, IndexBasedSpellCheckerTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, BasicZkTest, RecoveryZkTest, TestReplicationHandler, TestZkChroot, AutoCommitTest, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, TestRealTimeGet, TestJoin, SolrCmdDistributorTest, ConvertedLegacyTest, TestSort, BasicFunctionalityTest, TestBadConfig, TestIndexSearcher, HighlighterTest, ShowFileRequestHandlerTest, CurrencyFieldOpenExchangeTest, SolrIndexSplitterTest, SimplePostToolTest, CoreAdminHandlerTest, DocValuesTest, SuggesterTest, SchemaVersionSpecificBehaviorTest, SolrCoreCheckLockOnStartupTest, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, DebugComponentTest, TestSurroundQueryParser, DisMaxRequestHandlerTest, DocumentAnalysisRequestHandlerTest, TestOmitPositions, TermsComponentTest, XmlUpdateRequestHandlerTest, RequiredFieldsTest, TestArbitraryIndexDir, RegexBoostProcessorTest, TestJmxIntegration, UpdateRequestProcessorFactoryTest, TestCSVResponseWriter, QueryParsingTest, JsonLoaderTest, SOLR749Test, TestSolrIndexConfig, CopyFieldTest, TestDocSet, TestFuzzyAnalyzedSuggestions, TestPostingsSolrHighlighter, NotRequiredUniqueKeyTest, TestPhraseSuggestions, TestSweetSpotSimilarityFactory, TestLMDirichletSimilarityFactory, TestIBSimilarityFactory, TestDefaultSimilarityFactory, TestJmxMonitoredMap, ChaosMonkeyNothingIsSafeTest, DeleteInactiveReplicaTest, LeaderFailoverAfterPartitionTest, OutOfBoxZkACLAndCredentialsProvidersTest, RecoveryAfterSoftCommitTest]
   [junit4] Completed on J0 in 56.68s, 1 test, 1 failure <<< FAILURES!

[...truncated 299 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:529: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:477: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:61: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build.xml:191: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/common-build.xml:509: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:1370: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:976: There were test failures: 475 suites, 1893 tests, 1 error, 1 failure, 48 ignored (21 assumptions)

Total time: 46 minutes 48 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_60-ea-b06 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.8.0_60-ea-b06) - Build # 11904 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11904/
Java: 32bit/jdk1.8.0_60-ea-b06 -server -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Error from server at http://127.0.0.1:46515/m_xr/implicit_collection_without_routerfield_shard1_replica1: no servers hosting shard: 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:46515/m_xr/implicit_collection_without_routerfield_shard1_replica1: no servers hosting shard: 
	at __randomizedtesting.SeedInfo.seed([8E5F731690B17418:60B4CCC3E4D19E0]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:584)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:236)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:228)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:943)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:958)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDeleteByIdImplicitRouter(FullSolrCloudDistribCmdsTest.java:225)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:144)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:958)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:933)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 9758 lines...]
   [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
   [junit4]   2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/init-core-data-001
   [junit4]   2> 368750 T2269 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /m_xr/
   [junit4]   2> 368752 T2269 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 368753 T2270 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 368853 T2269 oasc.ZkTestServer.run start zk server on port:40052
   [junit4]   2> 368853 T2269 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 368854 T2269 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 368857 T2277 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19a494c name:ZooKeeperConnection Watcher:127.0.0.1:40052 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 368857 T2269 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 368859 T2269 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 368859 T2269 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 368866 T2269 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 368868 T2269 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 368869 T2280 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14bc2f name:ZooKeeperConnection Watcher:127.0.0.1:40052/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 368869 T2269 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 368870 T2269 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 368871 T2269 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 368873 T2269 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 368875 T2269 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 368876 T2269 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 368877 T2269 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 368878 T2269 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 368884 T2269 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 368885 T2269 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 368887 T2269 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 368888 T2269 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 368889 T2269 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 368889 T2269 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 368890 T2269 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 368891 T2269 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 368892 T2269 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 368893 T2269 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 368894 T2269 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 368895 T2269 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 368896 T2269 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 368897 T2269 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 368898 T2269 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 368898 T2269 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 368900 T2269 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 368900 T2269 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 368901 T2269 oasc.AbstractZkTestCase.putConfig put /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 368901 T2269 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 368973 T2269 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/cores/collection1
   [junit4]   2> 368974 T2269 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 368986 T2269 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55656
   [junit4]   2> 368991 T2269 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/tempDir-001/control/data, hostContext=/m_xr, hostPort=55656, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/cores}
   [junit4]   2> 368992 T2269 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@e2f2a
   [junit4]   2> 368993 T2269 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/'
   [junit4]   2> 369011 T2269 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/solr.xml
   [junit4]   2> 369015 T2269 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/cores
   [junit4]   2> 369016 T2269 oasc.CoreContainer.<init> New CoreContainer 16517148
   [junit4]   2> 369016 T2269 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/]
   [junit4]   2> 369016 T2269 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/lib
   [junit4]   2> 369016 T2269 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/lib).
   [junit4]   2> 369021 T2269 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 369021 T2269 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 369021 T2269 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 369022 T2269 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 369022 T2269 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 369022 T2269 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 369022 T2269 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 369022 T2269 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 369023 T2269 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 369023 T2269 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 369023 T2269 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 369023 T2269 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 369024 T2269 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 369024 T2269 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 369024 T2269 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 369024 T2269 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40052/solr
   [junit4]   2> 369025 T2269 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 369025 T2269 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 369025 T2269 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 369027 T2292 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1746944 name:ZooKeeperConnection Watcher:127.0.0.1:40052 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 369027 T2269 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 369028 T2269 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 369029 T2269 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 369033 T2295 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fde6b8 name:ZooKeeperConnection Watcher:127.0.0.1:40052/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 369033 T2269 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 369034 T2269 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 369035 T2269 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 369036 T2269 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 369037 T2269 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 369038 T2269 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 369039 T2269 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 369040 T2269 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 369041 T2269 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 369041 T2269 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55656_m_xr
   [junit4]   2> 369042 T2269 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55656_m_xr
   [junit4]   2> 369043 T2269 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 369043 T2269 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 369044 T2269 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 369045 T2269 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:55656_m_xr
   [junit4]   2> 369045 T2269 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 369046 T2269 oasc.Overseer.start Overseer (id=93533059765174275-127.0.0.1:55656_m_xr-n_0000000000) starting
   [junit4]   2> 369047 T2269 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 369049 T2269 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 369052 T2297 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 369052 T2296 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 369070 T2269 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 369077 T2269 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/cores
   [junit4]   2> 369078 T2269 oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/cores/collection1, collection=control_collection, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 369079 T2269 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/cores/collection1/
   [junit4]   2> 369079 T2269 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 369079 T2299 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 369079 T2299 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 369080 T2295 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 369080 T2299 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 369080 T2296 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55656/m_xr",
   [junit4]   2> 	  "node_name":"127.0.0.1:55656_m_xr",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 0
   [junit4]   2> 369081 T2296 oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55656/m_xr",
   [junit4]   2> 	  "node_name":"127.0.0.1:55656_m_xr",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 369081 T2296 oasco.ClusterStateMutator.createCollection building a new cName: control_collection
   [junit4]   2> 369081 T2296 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 369082 T2295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 370080 T2299 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 370081 T2299 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 370081 T2299 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 370081 T2299 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 370082 T2299 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 370083 T2299 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/cores/collection1/'
   [junit4]   2> 370096 T2299 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 370101 T2299 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 370115 T2299 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 370118 T2299 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.1.0
   [junit4]   2> 370169 T2299 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 370171 T2299 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 370177 T2299 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 370276 T2299 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 370279 T2299 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 370281 T2299 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 370286 T2299 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 370298 T2299 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 370298 T2299 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 370298 T2299 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/cores/collection1/, dataDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/cores/collection1/data/
   [junit4]   2> 370299 T2299 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1805333
   [junit4]   2> 370300 T2299 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/cores/collection1/data
   [junit4]   2> 370300 T2299 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/cores/collection1/data/index/
   [junit4]   2> 370300 T2299 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 370300 T2299 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/control-001/cores/collection1/data/index
   [junit4]   2> 370301 T2299 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=36, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 370301 T2299 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1eff318 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16b469c),segFN=segments_1,generation=1}
   [junit4]   2> 370302 T2299 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 370305 T2299 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 370306 T2299 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 370306 T2299 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 370306 T2299 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 370306 T2299 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 370307 T2299 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 370307 T2299 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 370307 T2299 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 370307 T2299 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 370309 T2299 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 370311 T2299 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 370312 T2299 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 370313 T2299 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 370317 T2299 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 370318 T2299 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 370319 T2299 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 370319 T2299 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
   [junit4]   2> 370320 T2299 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 370320 T2299 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 370321 T2299 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=64.8544921875, floorSegmentMB=0.544921875, forceMergeDeletesPctAllowed=25.4472438110377, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 370321 T2299 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1eff318 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16b469c),segFN=segments_1,generation=1}
   [junit4]   2> 370321 T2299 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 370322 T2299 oass.SolrIndexSearcher.<init> Opening Searcher@15464ec[collection1] main
   [junit4]   2> 370323 T2299 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 370323 T2299 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 370323 T2299 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 370324 T2299 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 370324 T2299 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 370324 T2299 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 370338 T2299 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 370339 T2299 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 370339 T2299 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 370339 T2299 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 370340 T2300 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@15464ec[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 370340 T2299 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 370345 T2303 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55656/m_xr collection:control_collection shard:shard1
   [junit4]   2> 370346 T2269 oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2
   [junit4]   2> 370347 T2269 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 370346 T2303 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 370347 T2269 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 370348 T2269 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 370348 T2306 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d21d83 name:ZooKeeperConnection Watcher:127.0.0.1:40052/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 370348 T2269 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 370349 T2269 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 370349 T2303 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 370349 T2269 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 370350 T2295 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 370350 T2303 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 370351 T2303 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 370351 T2269 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 370350 T2296 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"} current state version: 1
   [junit4]   2> 370351 T2269 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> ASYNC  NEW_CORE C332 name=collection1 org.apache.solr.core.SolrCore@b89946 url=http://127.0.0.1:55656/m_xr/collection1 node=127.0.0.1:55656_m_xr C332_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:55656/m_xr, node_name=127.0.0.1:55656_m_xr, state=down}
   [junit4]   2> 370351 T2303 C332 P55656 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55656/m_xr/collection1/
   [junit4]   2> 370352 T2269 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 370352 T2303 C332 P55656 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 370352 T2303 C332 P55656 oasc.SyncStrategy.syncToMe http://127.0.0.1:55656/m_xr/collection1/ has no replicas
   [junit4]   2> 370352 T2303 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55656/m_xr/collection1/ shard1
   [junit4]   2> 370352 T2303 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 370355 T2295 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 370356 T2296 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55656/m_xr",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 1
   [junit4]   2> 370357 T2269 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 370357 T2309 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ded493 name:ZooKeeperConnection Watcher:127.0.0.1:40052/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 370358 T2269 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 370359 T2269 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 370360 T2295 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 370361 T2296 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "stateFormat":"2"} current state version: 1
   [junit4]   2> 370361 T2296 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 370362 T2295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 370382 T2306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 370409 T2303 oasc.ZkController.register We are http://127.0.0.1:55656/m_xr/collection1/ and leader is http://127.0.0.1:55656/m_xr/collection1/
   [junit4]   2> 370409 T2303 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55656/m_xr
   [junit4]   2> 370409 T2303 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 370409 T2303 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 370410 T2303 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 370427 T2295 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 370427 T2296 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55656/m_xr",
   [junit4]   2> 	  "node_name":"127.0.0.1:55656_m_xr",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 370428 T2269 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/cores/collection1
   [junit4]   2> 370428 T2296 oasco.ReplicaMutator.updateState Update state numShards=3 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:55656/m_xr",
   [junit4]   2> 	  "node_name":"127.0.0.1:55656_m_xr",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 370429 T2269 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001
   [junit4]   2> 370429 T2269 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 370431 T2269 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46515
   [junit4]   2> 370431 T2269 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/m_xr, hostPort=46515, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/cores}
   [junit4]   2> 370432 T2269 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@e2f2a
   [junit4]   2> 370432 T2269 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/'
   [junit4]   2> 370441 T2296 oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/collection1/state.json
   [junit4]   2> 370443 T2306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 370443 T2295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 370444 T2269 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/solr.xml
   [junit4]   2> 370447 T2269 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/cores
   [junit4]   2> 370448 T2269 oasc.CoreContainer.<init> New CoreContainer 8136174
   [junit4]   2> 370448 T2269 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/]
   [junit4]   2> 370448 T2269 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/lib
   [junit4]   2> 370449 T2269 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/lib).
   [junit4]   2> 370455 T2269 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 370455 T2269 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 370456 T2269 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 370456 T2269 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 370456 T2269 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 370456 T2269 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 370456 T2269 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 370457 T2269 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 370457 T2269 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 370457 T2269 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 370457 T2269 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 370458 T2269 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 370458 T2269 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 370459 T2269 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 370459 T2269 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 370459 T2269 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40052/solr
   [junit4]   2> 370460 T2269 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 370460 T2269 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 370471 T2269 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 370471 T2321 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3466e0 name:ZooKeeperConnection Watcher:127.0.0.1:40052 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 370481 T2269 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 370482 T2269 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 370483 T2269 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 370484 T2324 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1463aeb name:ZooKeeperConnection Watcher:127.0.0.1:40052/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 370484 T2269 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 370493 T2269 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 370547 T2295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 370547 T2324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 370548 T2306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 371497 T2269 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46515_m_xr
   [junit4]   2> 371498 T2269 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46515_m_xr
   [junit4]   2> 371500 T2269 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 371501 T2269 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/cores
   [junit4]   2> 371502 T2269 oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/cores/collection1, collection=collection1, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 371503 T2269 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/cores/collection1/
   [junit4]   2> 371503 T2269 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 371504 T2325 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 371504 T2325 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 371506 T2295 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 371506 T2296 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46515/m_xr",
   [junit4]   2> 	  "node_name":"127.0.0.1:46515_m_xr",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 371507 T2296 oasco.ReplicaMutator.updateState Update state numShards=3 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46515/m_xr",
   [junit4]   2> 	  "node_name":"127.0.0.1:46515_m_xr",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 371507 T2296 oasco.ReplicaMutator.updateState Collection already exists with numShards=3
   [junit4]   2> 371507 T2296 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 371509 T2325 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 371509 T2325 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 371509 T2325 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 371511 T2325 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 
   [junit4]   2> 371511 T2325 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 371608 T2296 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 0
   [junit4]   2> 371610 T2324 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 2)
   [junit4]   2> 371611 T2324 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 
   [junit4]   2> 372511 T2325 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 372512 T2325 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 372512 T2325 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 372512 T2325 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 372513 T2325 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 372513 T2325 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/cores/collection1/'
   [junit4]   2> 372525 T2325 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 372526 T2325 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 372530 T2325 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 372535 T2325 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.1.0
   [junit4]   2> 372544 T2325 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 372546 T2325 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 372551 T2325 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 372700 T2325 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 372702 T2325 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 372703 T2325 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 372706 T2325 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 372730 T2325 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 372731 T2325 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 372731 T2325 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/cores/collection1/, dataDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/cores/collection1/data/
   [junit4]   2> 372731 T2325 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1805333
   [junit4]   2> 372732 T2325 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/cores/collection1/data
   [junit4]   2> 372732 T2325 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/cores/collection1/data/index/
   [junit4]   2> 372732 T2325 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 372733 T2325 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/../../../../../../../../../home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-1-001/cores/collection1/data/index
   [junit4]   2> 372733 T2325 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=36, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 372733 T2325 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@49b8a3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4c442c),segFN=segments_1,generation=1}
   [junit4]   2> 372735 T2325 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 372738 T2325 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 372738 T2325 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 372738 T2325 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 372738 T2325 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 372738 T2325 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 372739 T2325 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 372739 T2325 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 372739 T2325 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 372739 T2325 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 372740 T2325 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 372741 T2325 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 372741 T2325 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 372742 T2325 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 372746 T2325 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 372747 T2325 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 372747 T2325 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 372747 T2325 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
   [junit4]   2> 372748 T2325 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 372748 T2325 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 372751 T2325 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=64.8544921875, floorSegmentMB=0.544921875, forceMergeDeletesPctAllowed=25.4472438110377, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 372752 T2325 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@49b8a3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4c442c),segFN=segments_1,generation=1}
   [junit4]   2> 372752 T2325 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 372752 T2325 oass.SolrIndexSearcher.<init> Opening Searcher@1c680f9[collection1] main
   [junit4]   2> 372752 T2325 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 372753 T2325 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 372753 T2325 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 372753 T2325 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 372754 T2325 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 372754 T2325 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 372754 T2325 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 372754 T2325 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 372754 T2325 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 372755 T2325 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 372755 T2325 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 372756 T2326 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c680f9[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 372771 T2329 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46515/m_xr collection:collection1 shard:shard2
   [junit4]   2> 372771 T2269 oass.SolrDispatchFilter.init user.dir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2
   [junit4]   2> 372772 T2269 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 372772 T2329 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 372776 T2329 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 372777 T2295 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 372777 T2329 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 372777 T2329 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 372777 T2296 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"} current state version: 4
   [junit4]   2> ASYNC  NEW_CORE C333 name=collection1 org.apache.solr.core.SolrCore@15487f0 url=http://127.0.0.1:46515/m_xr/collection1 node=127.0.0.1:46515_m_xr C333_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:46515/m_xr, node_name=127.0.0.1:46515_m_xr, state=down}
   [junit4]   2> 372778 T2329 C333 P46515 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46515/m_xr/collection1/
   [junit4]   2> 372778 T2329 C333 P46515 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 372778 T2329 C333 P46515 oasc.SyncStrategy.syncToMe http://127.0.0.1:46515/m_xr/collection1/ has no replicas
   [junit4]   2> 372778 T2329 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46515/m_xr/collection1/ shard2
   [junit4]   2> 372778 T2296 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 1
   [junit4]   2> 372779 T2329 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 372779 T2324 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 2)
   [junit4]   2> 372780 T2324 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2 
   [junit4]   2> 372782 T2295 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 372783 T2296 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46515/m_xr",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 4
   [junit4]   2> 372783 T2296 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 2
   [junit4]   2> 372785 T2324 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 2)
   [junit4]   2> 372785 T2324 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 372833 T2329 oasc.ZkController.register We are http://127.0.0.1:46515/m_xr/collection1/ and leader is http://127.0.0.1:46515/m_xr/collection1/
   [junit4]   2> 372833 T2329 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46515/m_xr
   [junit4]   2> 372833 T2329 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 372833 T2329 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 372833 T2329 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 372835 T2295 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 372835 T2296 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46515/m_xr",
   [junit4]   2> 	  "node_name":"127.0.0.1:46515_m_xr",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 372836 T2296 oasco.ReplicaMutator.updateState Update state numShards=3 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46515/m_xr",
   [junit4]   2> 	  "node_name":"127.0.0.1:46515_m_xr",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 372839 T2329 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 372839 T2296 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 3
   [junit4]   2> 372840 T2324 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 2)
   [junit4]   2> 372840 T2324 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 372872 T2269 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/cores/collection1
   [junit4]   2> 372873 T2269 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2 in directory /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001
   [junit4]   2> 372874 T2269 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 372876 T2269 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44763
   [junit4]   2> 372877 T2269 oascse.JettySolrRunner$1.lifeCycleStarted Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/m_xr, hostPort=44763, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/cores}
   [junit4]   2> 372878 T2269 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@e2f2a
   [junit4]   2> 372878 T2269 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/'
   [junit4]   2> 372893 T2269 oasc.SolrXmlConfig.fromFile Loading container configuration from /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/solr.xml
   [junit4]   2> 372899 T2269 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/cores
   [junit4]   2> 372900 T2269 oasc.CoreContainer.<init> New CoreContainer 1348495
   [junit4]   2> 372900 T2269 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/]
   [junit4]   2> 372900 T2269 oasc.CoreContainer.load loading shared library: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/lib
   [junit4]   2> 372901 T2269 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/lib).
   [junit4]   2> 372909 T2269 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 372909 T2269 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 372910 T2269 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 372910 T2269 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 372910 T2269 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 372911 T2269 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 372911 T2269 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 372911 T2269 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 372912 T2269 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 372912 T2269 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 372912 T2269 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 372913 T2269 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 372913 T2269 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 372913 T2269 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 372914 T2269 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 372914 T2269 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40052/solr
   [junit4]   2> 372915 T2269 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 372915 T2269 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 372916 T2269 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 372917 T2341 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a2bbd7 name:ZooKeeperConnection Watcher:127.0.0.1:40052 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 372919 T2269 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 372920 T2269 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 372929 T2269 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 372930 T2344 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c4bd49 name:ZooKeeperConnection Watcher:127.0.0.1:40052/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 372930 T2269 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 372941 T2269 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 373959 T2269 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44763_m_xr
   [junit4]   2> 373960 T2269 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44763_m_xr
   [junit4]   2> 373962 T2269 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 373963 T2269 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/cores
   [junit4]   2> 373964 T2269 oasc.CoreDescriptor.<init> CORE DESCRIPTOR: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, instanceDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/cores/collection1, collection=collection1, absoluteInstDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/cores/collection1/, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 373965 T2269 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/cores/collection1/
   [junit4]   2> 373965 T2269 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 373971 T2345 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 373971 T2345 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 373972 T2295 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 373972 T2296 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44763/m_xr",
   [junit4]   2> 	  "node_name":"127.0.0.1:44763_m_xr",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 373972 T2345 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 373973 T2345 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 373973 T2296 oasco.ReplicaMutator.updateState Update state numShards=3 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44763/m_xr",
   [junit4]   2> 	  "node_name":"127.0.0.1:44763_m_xr",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 373973 T2345 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 373973 T2296 oasco.ReplicaMutator.updateState Collection already exists with numShards=3
   [junit4]   2> 373973 T2296 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 373973 T2345 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 373974 T2345 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 373975 T2296 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 4
   [junit4]   2> 373975 T2344 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 3)
   [junit4]   2> 373975 T2324 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json for collection collection1 has occurred - updating... (live nodes size: 3)
   [junit4]   2> 373976 T2344 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 373976 T2324 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 374974 T2345 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 374974 T2345 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 374975 T2345 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 374975 T2345 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 374975 T2345 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 374975 T2345 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/cores/collection1/'
   [junit4]   2> 374983 T2345 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 374984 T2345 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 374987 T2345 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 374989 T2345 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.1.0
   [junit4]   2> 374996 T2345 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 374996 T2345 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 374999 T2345 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 375079 T2345 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 375080 T2345 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 375081 T2345 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 375083 T2345 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 375090 T2345 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 375091 T2345 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 375091 T2345 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/cores/collection1/, dataDir=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/cores/collection1/data/
   [junit4]   2> 375091 T2345 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1805333
   [junit4]   2> 375092 T2345 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/cores/collection1/data
   [junit4]   2> 375092 T2345 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/cores/collection1/data/index/
   [junit4]   2> 375093 T2345 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 375093 T2345 oasc.CachingDirectoryFactory.get return new directory for /home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.FullSolrCloudDistribCmdsTest 8E5F731690B17418-001/shard-2-001/cores/collection1/data/index
   [junit4]   2> 375094 T2345 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=36, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 375094 T2345 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6e5949 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d04446),segFN=segments_1,generation=1}
   [junit4]   2> 375094 T2345 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 375097 T2345 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 375097 T2345 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 375097 T2345 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 375098 T2345 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 375098 T2345 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 375098 T2345 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 375098 T2345 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 375098 T2345 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 375099 T2345 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 375100 T2345 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 375100 T2345 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 375101 T2345 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 375102 T2345 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 375105 T2345 oasc.RequestHandlers.initHandlersFromConfig Registered paths: /admin/mbeans,standard,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 375105 T2345 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 375106 T2345 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 375106 T2345 oasu.UpdateLog.init Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10
   [junit4]   2> 375107 T2345 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 375107 T2345 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 375108 T2345 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=64.8544921875, floorSegmentMB=0.544921875, forceMergeDeletesPctAllowed=25.4472438110377, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 375108 T2345 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6e5949 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@d04446),segFN=segments_1,generation=1}
   [junit4]   2> 375108 T2345 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 375108 T2345 oass.SolrIndexSearcher.<init> Opening Searcher@137f09[collection1] main
   [junit4]   2> 375108 T2345 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 375109 T2345 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 375109 T2345 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 375109 T2345 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 375109 T2345 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 375109 T2345 oasr.Manage

[...truncated too long message...]

pFixedInterval), 5grand3_s=Lucene50(blocksize=128), 1parent_f1_s=PostingsFormat(name=MockRandom), 13parent_f1_s=Lucene50(blocksize=128), 23parent_f1_s=PostingsFormat(name=MockRandom), 6child1_s=PostingsFormat(name=MockRandom), _root_=Lucene50(blocksize=128), 5child2_s=PostingsFormat(name=MockRandom), 29parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 33parent_f1_s=Lucene50(blocksize=128), 39parent_f1_s=Lucene50(blocksize=128), 3child0_s=PostingsFormat(name=MockRandom), 4child3_s=PostingsFormat(name=MockRandom), 12parent_f1_s=PostingsFormat(name=MockRandom), 22parent_f1_s=Lucene50(blocksize=128), 1grand0_s=PostingsFormat(name=LuceneVarGapFixedInterval), 11grand2_s=Lucene50(blocksize=128), 11grand6_s=Lucene50(blocksize=128), 10grand7_s=Lucene50(blocksize=128), 7parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 2child1_s=PostingsFormat(name=MockRandom), 32parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 3parent_f2_s=Lucene50(blocksize=128), 10grand3_s=Lucene50(blocksize=128), 6grand3_s=PostingsFormat(name=LuceneVarGapFixedInterval), 31parent_f2_s=PostingsFormat(name=MockRandom), 5grand2_s=PostingsFormat(name=MockRandom), 10grand9_s=Lucene50(blocksize=128), 11parent_f2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 21parent_f2_s=Lucene50(blocksize=128), 4grand1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 7grand4_s=PostingsFormat(name=MockRandom), 3grand0_s=PostingsFormat(name=MockRandom), 9grand6_s=PostingsFormat(name=MockRandom), 8grand1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 9grand2_s=PostingsFormat(name=MockRandom), 8grand5_s=PostingsFormat(name=LuceneVarGapFixedInterval), 9parent_f1_s=PostingsFormat(name=MockRandom), 30parent_f1_s=PostingsFormat(name=MockRandom), 14parent_f2_s=Lucene50(blocksize=128), 20parent_f1_s=Lucene50(blocksize=128), 1parent_f2_s=Lucene50(blocksize=128), 6child3_s=PostingsFormat(name=LuceneVarGapFixedInterval), 7grand0_s=PostingsFormat(name=MockRandom), text_t=Lucene50(blocksize=128), 24parent_f2_s=PostingsFormat(name=MockRandom), 34parent_f2_s=Lucene50(blocksize=128), 13parent_f2_s=PostingsFormat(name=MockRandom), type_s=Lucene50(blocksize=128), 23parent_f2_s=Lucene50(blocksize=128), 5child1_s=Lucene50(blocksize=128), 10parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 33parent_f2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 11grand1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 22parent_f2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 10grand0_s=PostingsFormat(name=MockRandom), 4child0_s=Lucene50(blocksize=128), 11grand5_s=PostingsFormat(name=LuceneVarGapFixedInterval), 12parent_f2_s=Lucene50(blocksize=128), 10grand8_s=PostingsFormat(name=MockRandom), 10grand4_s=PostingsFormat(name=MockRandom), 32parent_f2_s=Lucene50(blocksize=128), 31parent_f1_s=Lucene50(blocksize=128), 5grand1_s=Lucene50(blocksize=128), 6grand4_s=Lucene50(blocksize=128), 11parent_f1_s=Lucene50(blocksize=128), 21parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 4grand2_s=Lucene50(blocksize=128), 7grand3_s=Lucene50(blocksize=128), 11grand9_s=PostingsFormat(name=LuceneVarGapFixedInterval), 8grand2_s=Lucene50(blocksize=128), 9grand5_s=Lucene50(blocksize=128), 8grand6_s=Lucene50(blocksize=128), 9grand1_s=Lucene50(blocksize=128), 37parent_f1_s=Lucene50(blocksize=128), 14parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 20parent_f2_s=PostingsFormat(name=MockRandom), 34parent_f1_s=PostingsFormat(name=MockRandom), 24parent_f1_s=Lucene50(blocksize=128), 6child4_s=Lucene50(blocksize=128), 30parent_f2_s=Lucene50(blocksize=128), 5parent_f1_s=PostingsFormat(name=MockRandom), 6grand0_s=Lucene50(blocksize=128), 10parent_f2_s=Lucene50(blocksize=128), 5child0_s=PostingsFormat(name=LuceneVarGapFixedInterval), 8parent_f1_s=Lucene50(blocksize=128), title_lettertok=PostingsFormat(name=MockRandom), 27parent_f2_s=Lucene50(blocksize=128), 2parent_f2_s=PostingsFormat(name=MockRandom), 11grand0_s=Lucene50(blocksize=128), 4child1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 2grand1_s=PostingsFormat(name=MockRandom), 11grand4_s=Lucene50(blocksize=128), 17parent_f2_s=PostingsFormat(name=MockRandom), 3child2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 16parent_f2_s=Lucene50(blocksize=128), 10grand5_s=Lucene50(blocksize=128), 26parent_f2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 1child0_s=PostingsFormat(name=LuceneVarGapFixedInterval), 36parent_f2_s=Lucene50(blocksize=128), 10grand1_s=Lucene50(blocksize=128), 7grand6_s=PostingsFormat(name=LuceneVarGapFixedInterval), 18parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 28parent_f1_s=Lucene50(blocksize=128), 5grand0_s=PostingsFormat(name=LuceneVarGapFixedInterval), 6grand5_s=PostingsFormat(name=MockRandom), 7grand2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 15parent_f1_s=Lucene50(blocksize=128), 4grand3_s=PostingsFormat(name=MockRandom), 9grand8_s=PostingsFormat(name=LuceneVarGapFixedInterval), 11grand8_s=Lucene50(blocksize=128), 8grand3_s=PostingsFormat(name=MockRandom), 6parent_f1_s=Lucene50(blocksize=128), 25parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 9grand4_s=PostingsFormat(name=LuceneVarGapFixedInterval), title=Lucene50(blocksize=128), 8grand7_s=PostingsFormat(name=MockRandom), 0parent_f1_s=Lucene50(blocksize=128), 3grand2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 35parent_f1_s=Lucene50(blocksize=128), 9grand0_s=PostingsFormat(name=LuceneVarGapFixedInterval), 19parent_f2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 4parent_f2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 37parent_f2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 5child4_s=PostingsFormat(name=LuceneVarGapFixedInterval), 6child5_s=PostingsFormat(name=MockRandom), 38parent_f1_s=PostingsFormat(name=MockRandom), 5grand4_s=PostingsFormat(name=LuceneVarGapFixedInterval), 6grand1_s=PostingsFormat(name=MockRandom), 5parent_f2_s=Lucene50(blocksize=128), 6child0_s=Lucene50(blocksize=128), 16parent_f1_s=PostingsFormat(name=MockRandom), 8parent_f2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 39parent_f2_s=PostingsFormat(name=MockRandom), 29parent_f2_s=Lucene50(blocksize=128), 2parent_f1_s=Lucene50(blocksize=128), 4child2_s=Lucene50(blocksize=128), rnd_b=PostingsFormat(name=MockRandom), 11grand3_s=PostingsFormat(name=MockRandom), 27parent_f1_s=PostingsFormat(name=MockRandom), 17parent_f1_s=Lucene50(blocksize=128), _version_=Lucene50(blocksize=128), 2grand0_s=Lucene50(blocksize=128), title_stemmed=Lucene50(blocksize=128), 11grand7_s=PostingsFormat(name=MockRandom), 3child1_s=Lucene50(blocksize=128), 10grand6_s=PostingsFormat(name=LuceneVarGapFixedInterval), 7parent_f2_s=Lucene50(blocksize=128), 26parent_f1_s=Lucene50(blocksize=128), 10grand2_s=PostingsFormat(name=LuceneVarGapFixedInterval), 2child0_s=Lucene50(blocksize=128), 3parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval), 36parent_f1_s=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=sq, timezone=Asia/Bangkok
   [junit4]   2> NOTE: Linux 3.13.0-46-generic i386/Oracle Corporation 1.8.0_60-ea (32-bit)/cpus=12,threads=1,free=160261632,total=342884352
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsSyncSliceTest, OverseerCollectionProcessorTest, QueryParsingTest, TestBM25SimilarityFactory, TestFieldSortValues, RegexBoostProcessorTest, SearchHandlerTest, DistributedFacetPivotSmallAdvancedTest, PolyFieldTest, BinaryUpdateRequestHandlerTest, BJQParserTest, UnloadDistributedZkTest, ConnectionManagerTest, DistributedQueryElevationComponentTest, TestSort, TestSolrXml, DistributedQueryComponentOptimizationTest, AtomicUpdatesTest, TestQueryTypes, QueryResultKeyTest, BasicDistributedZk2Test, SortByFunctionTest, TestCopyFieldCollectionResource, TestUtils, TestManagedStopFilterFactory, MergeStrategyTest, FacetPivotSmallTest, TestOverriddenPrefixQueryForCustomFieldType, TestShardHandlerFactory, SuggesterTest, TestGroupingSearch, DistributedExpandComponentTest, TestClassNameShortening, TestStressReorder, AssignTest, TermVectorComponentDistributedTest, NumericFieldsTest, TestCollationField, DocValuesTest, PrimUtilsTest, SpellCheckCollatorTest, TestDynamicFieldCollectionResource, TestSchemaSimilarityResource, CloudMLTQParserTest, AsyncMigrateRouteKeyTest, RecoveryZkTest, OutputWriterTest, TestObjectReleaseTracker, OverseerRolesTest, TestBulkSchemaAPI, TestReloadAndDeleteDocs, TestJsonFacets, TestLuceneMatchVersion, TestLeaderElectionZkExpiry, HdfsWriteToMultipleCollectionsTest, TestDocSet, TestSolrDynamicMBean, FullSolrCloudDistribCmdsTest]
   [junit4] Completed on J2 in 34.80s, 1 test, 1 error <<< FAILURES!

[...truncated 1014 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:529: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:477: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/build.xml:61: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/build.xml:191: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/solr/common-build.xml:509: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:1370: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-5.x-Linux/lucene/common-build.xml:976: There were test failures: 475 suites, 1892 tests, 1 error, 48 ignored (21 assumptions)

Total time: 43 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_60-ea-b06 -server -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any