You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/02/28 18:37:03 UTC

[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_51) - Build # 9519 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9519/
Java: 64bit/jdk1.7.0_51 -XX:+UseCompressedOops -XX:+UseSerialGC -XX:-UseSuperWord

1 tests failed.
FAILED:  org.apache.solr.client.solrj.impl.CloudSolrServerTest.testShutdown

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:39068 within 30000 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:39068 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([6326A836295DCF79:8050A1A3CE275A0B]:0)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:148)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:94)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:85)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:200)
	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:78)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:39068 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:222)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:140)
	... 47 more




Build Log:
[...truncated 11597 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
   [junit4]   2> 10483 T50 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 10484 T50 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /sm_gby/a
   [junit4]   2> 10486 T50 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
   [junit4]   2> 10504 T50 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1393608903188
   [junit4]   2> 10508 T50 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 10511 T51 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 10657 T50 oasc.ZkTestServer.run start zk server on port:39068
   [junit4]   2> 10690 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 10700 T52 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 45257 T54 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 32939ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 45280 T52 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14479904a4b0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 45282 T52 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14479904a4b0001, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 45285 T57 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71284e3 name:ZooKeeperConnection Watcher:127.0.0.1:39068 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 45286 T57 oascc.ConnectionManager.process Client->ZooKeeper status change trigger but we are already closed
   [junit4]   2> 45292 T50 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
   [junit4]   2> 45296 T50 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:39068 39068
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrServerTest -Dtests.method=testShutdown -Dtests.seed=6326A836295DCF79 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=vi_VN -Dtests.timezone=Pacific/Marquesas -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   40.0s J0 | CloudSolrServerTest.testShutdown <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:39068 within 30000 ms
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([6326A836295DCF79:8050A1A3CE275A0B]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:148)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:94)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:85)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:200)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:78)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:39068 within 30000 ms
   [junit4]    > 	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:222)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:140)
   [junit4]    > 	... 47 more
   [junit4]   2> 50513 T50 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1393608943196
   [junit4]   2> 50513 T50 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 50514 T58 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 50614 T50 oasc.ZkTestServer.run start zk server on port:49340
   [junit4]   2> 50615 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 50618 T64 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@378f5b14 name:ZooKeeperConnection Watcher:127.0.0.1:49340 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 50618 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 50620 T50 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 50642 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 50643 T66 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@330fd2da name:ZooKeeperConnection Watcher:127.0.0.1:49340/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 50644 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 50648 T50 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 50653 T50 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 50659 T50 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 50669 T50 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 50674 T50 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 50685 T50 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 50691 T50 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 50692 T50 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 50702 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 50703 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
   [junit4]   2> 50703 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
   [junit4]   2> 50703 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
   [junit4]   2> 50704 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 50704 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 50705 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 50705 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 50705 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 50763 T50 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 50792 T50 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33205
   [junit4]   2> 50793 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 50793 T50 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 50793 T50 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393608943402
   [junit4]   2> 50794 T50 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393608943402/'
   [junit4]   2> 50806 T50 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393608943402/solr.xml
   [junit4]   2> 50858 T50 oasc.CoreContainer.<init> New CoreContainer 1700712206
   [junit4]   2> 50858 T50 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393608943402/]
   [junit4]   2> 50859 T50 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 50860 T50 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 50860 T50 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 50861 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 50861 T50 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 50862 T50 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 50862 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 50863 T50 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 50864 T50 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 50868 T50 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 50868 T50 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 50869 T50 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 50869 T50 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49340/solr
   [junit4]   2> 50882 T50 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 50883 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 50887 T77 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a22c38f name:ZooKeeperConnection Watcher:127.0.0.1:49340 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 50888 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 50893 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 50896 T79 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74845058 name:ZooKeeperConnection Watcher:127.0.0.1:49340/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 50896 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 50901 T50 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 50909 T50 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 50919 T50 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 50922 T50 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33205_sm_gby%2Fa
   [junit4]   2> 50926 T50 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33205_sm_gby%2Fa
   [junit4]   2> 50932 T50 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 50936 T50 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 50946 T50 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:33205_sm_gby%2Fa
   [junit4]   2> 50947 T50 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 50951 T50 oasc.Overseer.start Overseer (id=91331555701555203-127.0.0.1:33205_sm_gby%2Fa-n_0000000000) starting
   [junit4]   2> 50964 T50 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 51018 T81 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 51019 T50 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 51023 T50 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 51026 T50 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 51028 T81 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 51035 T80 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 51042 T82 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 51043 T82 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 51046 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 51047 T82 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 51050 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 51052 T80 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:33205/sm_gby/a",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:33205_sm_gby%2Fa",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 51052 T80 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 51059 T80 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 51064 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 51068 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 52047 T82 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 52048 T82 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393608943402/collection1
   [junit4]   2> 52049 T82 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 52050 T82 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 52050 T82 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 52053 T82 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 52053 T82 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393608943402/collection1/'
   [junit4]   2> 52113 T82 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 52166 T82 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 52169 T82 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 52230 T82 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 52301 T82 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableIntField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 52311 T82 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableLongField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 52321 T82 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableFloatField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 52331 T82 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableDoubleField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 52709 T82 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 52712 T82 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 52715 T82 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 52715 T82 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 52720 T82 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 52720 T82 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393608943402/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/control/data/
   [junit4]   2> 52720 T82 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 52721 T82 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/control/data
   [junit4]   2> 52721 T82 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/control/data/index/
   [junit4]   2> 52722 T82 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 52722 T82 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/control/data/index
   [junit4]   2> 52723 T82 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@52d6c570 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 52724 T82 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 52725 T82 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 52725 T82 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 52725 T82 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 52726 T82 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 52729 T82 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 52730 T82 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 52730 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 52730 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 52732 T82 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 52735 T82 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 52735 T82 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 52736 T82 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@52d6c570 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 52737 T82 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 52737 T82 oass.SolrIndexSearcher.<init> Opening Searcher@614b722[collection1] main
   [junit4]   2> 52738 T83 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@614b722[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 52738 T82 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 52740 T50 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 52741 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 52740 T86 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33205/sm_gby/a collection:control_collection shard:shard1
   [junit4]   2> 52747 T86 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 52749 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 52751 T88 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29f8f02c name:ZooKeeperConnection Watcher:127.0.0.1:49340/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 52751 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 52756 T50 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 52769 T86 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 52771 T50 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 52777 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 52784 T86 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 52784 T86 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 52785 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C6 name=collection1 org.apache.solr.core.SolrCore@488b250 url=http://127.0.0.1:33205/sm_gby/a/collection1 node=127.0.0.1:33205_sm_gby%2Fa C6_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:33205/sm_gby/a, core=collection1, node_name=127.0.0.1:33205_sm_gby%2Fa}
   [junit4]   2> 52786 T86 C6 P33205 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33205/sm_gby/a/collection1/
   [junit4]   2> 52787 T86 C6 P33205 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 52787 T86 C6 P33205 oasc.SyncStrategy.syncToMe http://127.0.0.1:33205/sm_gby/a/collection1/ has no replicas
   [junit4]   2> 52787 T86 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33205/sm_gby/a/collection1/ shard1
   [junit4]   2> 52788 T86 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 52793 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 52794 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 52794 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 52810 T50 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 52810 T50 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 52813 T50 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54864
   [junit4]   2> 52814 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 52814 T50 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 52815 T50 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393608945454
   [junit4]   2> 52815 T50 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393608945454/'
   [junit4]   2> 52827 T50 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393608945454/solr.xml
   [junit4]   2> 52873 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 52874 T50 oasc.CoreContainer.<init> New CoreContainer 1597724868
   [junit4]   2> 52875 T50 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393608945454/]
   [junit4]   2> 52876 T50 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 52876 T50 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 52877 T50 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 52877 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 52877 T50 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 52878 T50 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 52878 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 52878 T50 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 52879 T50 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 52880 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 52884 T50 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 52885 T50 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 52885 T50 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 52886 T50 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49340/solr
   [junit4]   2> 52886 T50 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 52888 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 52892 T99 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d9e8821 name:ZooKeeperConnection Watcher:127.0.0.1:49340 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 52892 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 52896 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 52899 T101 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ede9641 name:ZooKeeperConnection Watcher:127.0.0.1:49340/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 52899 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 52906 T50 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 52984 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 52984 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 52984 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 53023 T86 oasc.ZkController.register We are http://127.0.0.1:33205/sm_gby/a/collection1/ and leader is http://127.0.0.1:33205/sm_gby/a/collection1/
   [junit4]   2> 53023 T86 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33205/sm_gby/a
   [junit4]   2> 53023 T86 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 53024 T86 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 53024 T86 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 53025 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 53026 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 53026 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 53026 T86 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 53028 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 53030 T80 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:33205/sm_gby/a",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:33205_sm_gby%2Fa",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 53034 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 53137 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 53137 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 53138 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 53911 T50 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54864_sm_gby%2Fa
   [junit4]   2> 53913 T50 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54864_sm_gby%2Fa
   [junit4]   2> 53918 T101 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 53918 T79 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 53919 T88 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 53934 T102 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 53934 T102 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 53937 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 53937 T102 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 53937 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 53938 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 53940 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 53942 T80 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54864/sm_gby/a",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54864_sm_gby%2Fa",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 53942 T80 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 53943 T80 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 53949 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 54053 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 54053 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 54054 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 54938 T102 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 54939 T102 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393608945454/collection1
   [junit4]   2> 54939 T102 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 54940 T102 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 54940 T102 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 54942 T102 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 54942 T102 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393608945454/collection1/'
   [junit4]   2> 54976 T102 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 55018 T102 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 55021 T102 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 55070 T102 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 55453 T102 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 55456 T102 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 55458 T102 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 55459 T102 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 55460 T102 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 55460 T102 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393608945454/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty1/
   [junit4]   2> 55461 T102 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 55461 T102 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty1
   [junit4]   2> 55462 T102 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty1/index/
   [junit4]   2> 55462 T102 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 55462 T102 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty1/index
   [junit4]   2> 55464 T102 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@38516a27 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 55464 T102 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 55465 T102 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 55465 T102 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 55466 T102 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 55466 T102 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 55466 T102 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 55467 T102 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 55467 T102 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 55467 T102 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 55469 T102 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 55472 T102 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 55473 T102 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 55474 T102 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@38516a27 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 55474 T102 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 55474 T102 oass.SolrIndexSearcher.<init> Opening Searcher@5421641[collection1] main
   [junit4]   2> 55475 T103 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5421641[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 55475 T102 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 55476 T50 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 55477 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 55476 T106 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54864/sm_gby/a collection:collection1 shard:shard2
   [junit4]   2> 55479 T106 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 55490 T106 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 55493 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55493 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55494 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55494 T106 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 55494 T106 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C7 name=collection1 org.apache.solr.core.SolrCore@4c112d16 url=http://127.0.0.1:54864/sm_gby/a/collection1 node=127.0.0.1:54864_sm_gby%2Fa C7_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:54864/sm_gby/a, core=collection1, node_name=127.0.0.1:54864_sm_gby%2Fa}
   [junit4]   2> 55495 T106 C7 P54864 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54864/sm_gby/a/collection1/
   [junit4]   2> 55495 T106 C7 P54864 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 55495 T106 C7 P54864 oasc.SyncStrategy.syncToMe http://127.0.0.1:54864/sm_gby/a/collection1/ has no replicas
   [junit4]   2> 55495 T106 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54864/sm_gby/a/collection1/ shard2
   [junit4]   2> 55496 T106 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 55496 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 55501 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55506 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55508 T50 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 55508 T50 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 55511 T50 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49022
   [junit4]   2> 55512 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55512 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 55513 T50 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 55513 T50 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393608948160
   [junit4]   2> 55513 T50 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393608948160/'
   [junit4]   2> 55526 T50 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393608948160/solr.xml
   [junit4]   2> 55581 T50 oasc.CoreContainer.<init> New CoreContainer 853927167
   [junit4]   2> 55581 T50 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393608948160/]
   [junit4]   2> 55583 T50 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 55583 T50 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 55584 T50 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 55584 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 55585 T50 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 55585 T50 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 55586 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 55586 T50 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 55587 T50 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 55590 T50 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 55591 T50 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 55591 T50 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 55591 T50 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49340/solr
   [junit4]   2> 55592 T50 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 55592 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 55594 T117 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@131d3b74 name:ZooKeeperConnection Watcher:127.0.0.1:49340 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 55595 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 55598 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 55599 T119 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70c18d7f name:ZooKeeperConnection Watcher:127.0.0.1:49340/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 55600 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 55606 T50 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 55646 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 55646 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 55646 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 55646 T119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 55657 T106 oasc.ZkController.register We are http://127.0.0.1:54864/sm_gby/a/collection1/ and leader is http://127.0.0.1:54864/sm_gby/a/collection1/
   [junit4]   2> 55657 T106 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54864/sm_gby/a
   [junit4]   2> 55658 T106 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 55658 T106 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 55658 T106 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 55661 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55661 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55661 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55661 T106 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 55663 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 55665 T80 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54864/sm_gby/a",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54864_sm_gby%2Fa",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 55669 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 55772 T119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 55772 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 55772 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 55772 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 56612 T50 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49022_sm_gby%2Fa
   [junit4]   2> 56615 T50 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49022_sm_gby%2Fa
   [junit4]   2> 56622 T88 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 56622 T79 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 56623 T119 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 56624 T101 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 56643 T120 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 56643 T120 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 56646 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 56646 T120 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 56646 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 56647 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 56652 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 56655 T80 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49022/sm_gby/a",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49022_sm_gby%2Fa",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 56656 T80 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 56656 T80 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 56663 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 56769 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 56769 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 56769 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 56769 T119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 57647 T120 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 57648 T120 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393608948160/collection1
   [junit4]   2> 57648 T120 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 57649 T120 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 57649 T120 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 57651 T120 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 57651 T120 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393608948160/collection1/'
   [junit4]   2> 57700 T120 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 57739 T120 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 57741 T120 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 57770 T120 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 58148 T120 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 58151 T120 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 58153 T120 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 58153 T120 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 58154 T120 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 58155 T120 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393608948160/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty2/
   [junit4]   2> 58155 T120 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 58156 T120 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty2
   [junit4]   2> 58156 T120 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty2/index/
   [junit4]   2> 58156 T120 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 58156 T120 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty2/index
   [junit4]   2> 58158 T120 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2c82719b lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 58158 T120 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 58159 T120 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 58159 T120 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 58160 T120 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 58160 T120 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 58160 T120 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 58161 T120 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 58161 T120 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 58161 T120 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 58163 T120 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 58166 T120 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 58167 T120 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 58167 T120 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2c82719b lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 58168 T120 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 58168 T120 oass.SolrIndexSearcher.<init> Opening Searcher@79c1bab1[collection1] main
   [junit4]   2> 58169 T121 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79c1bab1[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 58169 T120 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 58171 T50 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 58171 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 58171 T124 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49022/sm_gby/a collection:collection1 shard:shard1
   [junit4]   2> 58173 T124 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 58181 T124 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 58184 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58185 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58185 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58186 T124 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 58186 T124 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C8 name=collection1 org.apache.solr.core.SolrCore@66c47c26 url=http://127.0.0.1:49022/sm_gby/a/collection1 node=127.0.0.1:49022_sm_gby%2Fa C8_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:49022/sm_gby/a, core=collection1, node_name=127.0.0.1:49022_sm_gby%2Fa}
   [junit4]   2> 58186 T124 C8 P49022 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49022/sm_gby/a/collection1/
   [junit4]   2> 58188 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 58188 T124 C8 P49022 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 58189 T124 C8 P49022 oasc.SyncStrategy.syncToMe http://127.0.0.1:49022/sm_gby/a/collection1/ has no replicas
   [junit4]   2> 58190 T124 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49022/sm_gby/a/collection1/ shard1
   [junit4]   2> 58190 T124 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 58193 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58201 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58203 T50 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 58204 T50 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 58206 T50 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52027
   [junit4]   2> 58207 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 58208 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58208 T50 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 58209 T50 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393608950855
   [junit4]   2> 58209 T50 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393608950855/'
   [junit4]   2> 58229 T50 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393608950855/solr.xml
   [junit4]   2> 58273 T50 oasc.CoreContainer.<init> New CoreContainer 1956333402
   [junit4]   2> 58273 T50 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393608950855/]
   [junit4]   2> 58274 T50 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 58275 T50 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 58275 T50 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 58275 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 58275 T50 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 58276 T50 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 58276 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 58276 T50 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 58277 T50 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 58280 T50 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 58281 T50 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 58281 T50 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 58281 T50 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49340/solr
   [junit4]   2> 58282 T50 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 58283 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 58285 T135 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e69a184 name:ZooKeeperConnection Watcher:127.0.0.1:49340 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 58286 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 58289 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 58291 T137 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c710710 name:ZooKeeperConnection Watcher:127.0.0.1:49340/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 58292 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 58299 T50 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 58336 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58337 T137 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58336 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58336 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58336 T119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58353 T124 oasc.ZkController.register We are http://127.0.0.1:49022/sm_gby/a/collection1/ and leader is http://127.0.0.1:49022/sm_gby/a/collection1/
   [junit4]   2> 58354 T124 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49022/sm_gby/a
   [junit4]   2> 58354 T124 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 58354 T124 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 58354 T124 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 58357 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58358 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58358 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58359 T124 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 58361 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 58363 T80 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49022/sm_gby/a",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49022_sm_gby%2Fa",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 58366 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 58472 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58472 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58472 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58472 T119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 58472 T137 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 59306 T50 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52027_sm_gby%2Fa
   [junit4]   2> 59307 T50 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52027_sm_gby%2Fa
   [junit4]   2> 59312 T101 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 59313 T79 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 59312 T88 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 59313 T137 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 59313 T119 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 59324 T138 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 59324 T138 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 59326 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 59327 T138 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 59327 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 59327 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 59329 T80 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 59331 T80 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52027/sm_gby/a",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52027_sm_gby%2Fa",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 59331 T80 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 59331 T80 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 59335 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 59438 T79 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 59438 T88 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 59439 T137 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 59439 T119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 59439 T101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 60327 T138 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 60328 T138 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393608950855/collection1
   [junit4]   2> 60328 T138 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 60329 T138 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 60330 T138 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 60331 T138 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 60332 T138 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393608950855/collection1/'
   [junit4]   2> 60379 T138 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 60419 T138 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 60421 T138 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 60448 T138 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 60801 T138 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 60804 T138 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 60806 T138 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 60806 T138 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 60807 T138 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 60807 T138 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393608950855/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/
   [junit4]   2> 60808 T138 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 60809 T138 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3
   [junit4]   2> 60809 T138 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/index/
   [junit4]   2> 60809 T138 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 60809 T138 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/index
   [junit4]   2> 60811 T138 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6d66cc49 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/index),segFN=segments_1,generation=1}
   [junit4]   2> 60811 T138 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 60812 T138 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 60813 T138 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 60813 T138 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 60813 T138 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 60813 T138 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 60814 T138 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 60814 T138 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 60814 T138 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 60817 T138 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 60819 T138 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 60820 T138 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 60820 T138 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6d66cc49 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/index),segFN=segments_1,generation=1}
   [junit4]   2> 60821 T138 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 60821 T138 oass.SolrIndexSearcher.<init> Opening Searcher@55c9e26f[collection1] main
   [junit4]   2> 60822 T139 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@55c9e26f[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 60822 T138 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 60824 T50 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 60824 T142 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52027/sm_gby/a collection:collection1 shard:shard2
   [junit4]   2> 60824 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 60835 T142 oasc.ZkController.register We are http://127.0.0.1:52027/sm_gby/a/collection1/ and leader is http://127.0.0.1:54864/sm_gby/a/collection1/
   [junit4]   2> 60835 T142 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52027/sm_gby/a
   [junit4]   2> 60835 T142 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 60836 T142 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C9 name=collection1 org.apache.solr.core.SolrCore@48ccb726 url=http://127.0.0.1:52027/sm_gby/a/collection1 node=127.0.0.1:52027_sm_gby%2Fa C9_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:52027/sm_gby/a, core=collection1, node_name=127.0.0.1:52027_sm_gby%2Fa}
   [junit4]   2> 60841 T143 C9 P52027 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 60843 T143 C9 P52027 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 60843 T142 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 60844 T143 C9 P52027 oasc.ZkController.publish publishing core=collection1 state=recovering
   [junit4]   2> 60844 T143 C9 P52027 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 60846 T79 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 60846 T79 oasc.DistributedQueue$LatchChildWa

[...truncated too long message...]

tate=active, base_url=http://127.0.0.1:52027/sm_gby/a, core=collection1, node_name=127.0.0.1:52027_sm_gby%2Fa}
   [junit4]   2> 72790 T137 C21 P52027 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 71867 T119 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 71778 T101 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 80775 T101 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 80781 T156 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 80781 T156 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6c9d1e9b
   [junit4]   2> 80781 T137 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 80782 T137 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@48ccb726
   [junit4]   2> 80783 T156 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=1,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=5}
   [junit4]   2> 80784 T50 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 80785 T137 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=1,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=5}
   [junit4]   2> 80784 T156 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 80786 T156 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 80786 T156 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 80785 T137 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 80787 T137 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 80788 T137 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 80789 T156 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 80790 T156 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 80790 T156 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty4;done=false>>]
   [junit4]   2> 80791 T137 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 80791 T156 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty4
   [junit4]   2> 80791 T137 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 80792 T156 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty4/index;done=false>>]
   [junit4]   2> 80792 T156 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty4/index
   [junit4]   2> 80792 T137 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/index;done=false>>]
   [junit4]   2> 80794 T137 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3/index
   [junit4]   2> 80793 T156 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 80794 T137 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3;done=false>>]
   [junit4]   2> 80795 T156 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 80795 T137 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393608943196/jetty3
   [junit4]   2> 80797 T137 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 80797 T137 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 80798 T137 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 80799 T137 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=vi_VN, timezone=Pacific/Marquesas
   [junit4]   2> NOTE: Linux 3.8.0-36-generic amd64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=8,threads=1,free=92268280,total=126484480
   [junit4]   2> NOTE: All tests run in this JVM: [LargeVolumeJettyTest, LargeVolumeEmbeddedTest, TestDocumentObjectBinder, CloudSolrServerTest]
   [junit4] Completed on J0 in 71.42s, 2 tests, 1 error <<< FAILURES!

[...truncated 109 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:451: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1278: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:911: There were test failures: 48 suites, 280 tests, 1 error

Total time: 48 minutes 17 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_51 -XX:+UseCompressedOops -XX:+UseSerialGC -XX:-UseSuperWord
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0-fcs-b129) - Build # 9520 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9520/
Java: 32bit/jdk1.8.0-fcs-b129 -server -XX:+UseConcMarkSweepGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ShardSplitTest.testDistribSearch

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:47455

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:47455
	at __randomizedtesting.SeedInfo.seed([B21111266F3E552:8AC79F0A11AC856E]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:507)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
	at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:536)
	at org.apache.solr.cloud.ShardSplitTest.incompleteOrOverlappingCustomRangeTest(ShardSplitTest.java:159)
	at org.apache.solr.cloud.ShardSplitTest.doTest(ShardSplitTest.java:111)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:150)
	at java.net.SocketInputStream.read(SocketInputStream.java:121)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:395)
	... 45 more




Build Log:
[...truncated 10275 lines...]
   [junit4] Suite: org.apache.solr.cloud.ShardSplitTest
   [junit4]   2> 16851 T48 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 16864 T48 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-ShardSplitTest-1393614572244
   [junit4]   2> 16867 T48 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 16869 T49 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 16969 T48 oasc.ZkTestServer.run start zk server on port:42702
   [junit4]   2> 17030 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 17046 T50 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 18597 T55 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@293765 name:ZooKeeperConnection Watcher:127.0.0.1:42702 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 18598 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 18599 T48 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 18626 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 18628 T57 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16b7080 name:ZooKeeperConnection Watcher:127.0.0.1:42702/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 18628 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 18634 T48 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 18638 T48 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 18641 T48 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 18644 T48 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 18647 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 18657 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 18662 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 18663 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 18668 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 18669 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 18672 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 18673 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 18676 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 18676 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 18680 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 18680 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 18683 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 18684 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 18687 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 18687 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 18691 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 18691 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 18694 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 18695 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 18698 T48 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 18699 T48 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 19081 T48 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 19161 T48 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41673
   [junit4]   2> 19201 T48 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 19202 T48 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 19202 T48 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084
   [junit4]   2> 19202 T48 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/'
   [junit4]   2> 19212 T48 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/solr.xml
   [junit4]   2> 19224 T48 oasc.CoreContainer.<init> New CoreContainer 1824674
   [junit4]   2> 19224 T48 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/]
   [junit4]   2> 19234 T48 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 19235 T48 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 19235 T48 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 19235 T48 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 19236 T48 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 19236 T48 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 19236 T48 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 19236 T48 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 19237 T48 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 19239 T48 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 19239 T48 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 19240 T48 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 19240 T48 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42702/solr
   [junit4]   2> 19249 T48 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 19250 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 19253 T69 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1158d81 name:ZooKeeperConnection Watcher:127.0.0.1:42702 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 19254 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 19262 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 19266 T71 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48654d name:ZooKeeperConnection Watcher:127.0.0.1:42702/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 19266 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 19274 T48 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 19283 T48 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 19294 T48 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 19297 T48 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41673_
   [junit4]   2> 19301 T48 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41673_
   [junit4]   2> 19307 T48 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 19309 T48 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 19318 T48 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:41673_
   [junit4]   2> 19318 T48 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 19321 T48 oasc.Overseer.start Overseer (id=91331924614250499-127.0.0.1:41673_-n_0000000000) starting
   [junit4]   2> 19331 T48 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 19344 T73 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 19344 T48 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 19347 T48 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 19351 T48 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 19353 T73 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 19360 T72 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 19365 T74 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 19365 T74 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 19371 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 19371 T74 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 19374 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 19375 T72 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41673",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:41673_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 19375 T72 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 19382 T72 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 19388 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 19392 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 20372 T74 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 20372 T74 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/collection1
   [junit4]   2> 20372 T74 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 20373 T74 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 20374 T74 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 20375 T74 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 20375 T74 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/collection1/'
   [junit4]   2> 20376 T74 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/collection1/lib/.svn/' to classloader
   [junit4]   2> 20376 T74 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/collection1/lib/classes/' to classloader
   [junit4]   2> 20377 T74 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/collection1/lib/README' to classloader
   [junit4]   2> 20409 T74 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 20423 T74 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 20425 T74 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 20433 T74 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 20624 T74 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 20625 T74 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 20626 T74 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 20629 T74 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 20660 T74 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 20661 T74 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ShardSplitTest-controljetty-1393614574084/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1393614572244/control/data/
   [junit4]   2> 20661 T74 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@142865a
   [junit4]   2> 20665 T74 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1393614572244/control/data
   [junit4]   2> 20666 T74 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1393614572244/control/data/index/
   [junit4]   2> 20666 T74 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1393614572244/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 20666 T74 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1393614572244/control/data/index
   [junit4]   2> 20667 T74 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=17, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7989922721774966]
   [junit4]   2> 20667 T74 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@101186d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ad24e0),segFN=segments_1,generation=1}
   [junit4]   2> 20668 T74 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 20673 T74 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 20684 T74 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 20684 T74 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 20685 T74 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 20685 T74 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 20685 T74 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 20685 T74 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 20686 T74 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 20686 T74 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 20688 T74 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 20689 T74 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 20689 T74 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 20689 T74 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 20690 T74 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 20690 T74 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 20691 T74 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 20693 T74 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 20694 T74 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 20694 T74 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 20696 T74 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=9.20703125, floorSegmentMB=1.8662109375, forceMergeDeletesPctAllowed=5.930420212113585, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4004882394888162
   [junit4]   2> 20696 T74 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@101186d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ad24e0),segFN=segments_1,generation=1}
   [junit4]   2> 20697 T74 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 20697 T74 oass.SolrIndexSearcher.<init> Opening Searcher@19e5fa5[collection1] main
   [junit4]   2> 20703 T75 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19e5fa5[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 20707 T74 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 20708 T78 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41673 collection:control_collection shard:shard1
   [junit4]   2> 20708 T48 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 20708 T48 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 20715 T78 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 20723 T78 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 20727 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 20728 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20729 T78 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 20729 T80 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14dab45 name:ZooKeeperConnection Watcher:127.0.0.1:42702/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 20729 T78 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 20729 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> ASYNC  NEW_CORE C40 name=collection1 org.apache.solr.core.SolrCore@872f43 url=http://127.0.0.1:41673/collection1 node=127.0.0.1:41673_ C40_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:41673, core=collection1, node_name=127.0.0.1:41673_}
   [junit4]   2> 20730 T78 C40 P41673 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:41673/collection1/
   [junit4]   2> 20730 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 20731 T78 C40 P41673 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 20731 T78 C40 P41673 oasc.SyncStrategy.syncToMe http://127.0.0.1:41673/collection1/ has no replicas
   [junit4]   2> 20731 T78 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:41673/collection1/ shard1
   [junit4]   2> 20731 T78 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 20733 T48 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 20734 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20741 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20744 T48 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 20749 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20853 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 20856 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 20893 T78 oasc.ZkController.register We are http://127.0.0.1:41673/collection1/ and leader is http://127.0.0.1:41673/collection1/
   [junit4]   2> 20894 T78 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41673
   [junit4]   2> 20894 T78 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 20894 T78 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 20894 T78 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 20895 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20895 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20896 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20896 T78 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 20897 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 20898 T72 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41673",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:41673_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 20901 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 20995 T48 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 20996 T48 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 20999 T48 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36156
   [junit4]   2> 21000 T48 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 21000 T48 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 21000 T48 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124
   [junit4]   2> 21000 T48 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/'
   [junit4]   2> 21003 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 21003 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 21010 T48 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/solr.xml
   [junit4]   2> 21025 T48 oasc.CoreContainer.<init> New CoreContainer 22746881
   [junit4]   2> 21026 T48 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/]
   [junit4]   2> 21026 T48 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 21027 T48 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 21027 T48 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 21027 T48 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 21028 T48 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 21028 T48 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 21028 T48 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 21029 T48 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 21029 T48 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 21032 T48 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 21032 T48 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 21032 T48 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 21033 T48 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42702/solr
   [junit4]   2> 21033 T48 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 21034 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 21038 T91 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15d96c0 name:ZooKeeperConnection Watcher:127.0.0.1:42702 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 21039 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 21043 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 21045 T93 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@796175 name:ZooKeeperConnection Watcher:127.0.0.1:42702/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 21045 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 21049 T48 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 22052 T48 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36156_
   [junit4]   2> 22054 T48 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36156_
   [junit4]   2> 22057 T71 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 22057 T93 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 22057 T80 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 22062 T94 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 22062 T94 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 22063 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 22063 T94 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 22063 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 22064 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 22065 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 22066 T72 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36156",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36156_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 22066 T72 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 22066 T72 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 22068 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 22173 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 22173 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 22173 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 23064 T94 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 23064 T94 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/collection1
   [junit4]   2> 23064 T94 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 23065 T94 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 23065 T94 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 23068 T94 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 23068 T94 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/collection1/'
   [junit4]   2> 23069 T94 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/collection1/lib/.svn/' to classloader
   [junit4]   2> 23069 T94 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/collection1/lib/classes/' to classloader
   [junit4]   2> 23070 T94 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/collection1/lib/README' to classloader
   [junit4]   2> 23100 T94 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 23116 T94 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 23118 T94 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 23125 T94 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 23305 T94 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 23307 T94 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 23308 T94 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 23313 T94 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 23365 T94 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 23366 T94 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ShardSplitTest-jetty1-1393614576124/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty1/
   [junit4]   2> 23366 T94 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@142865a
   [junit4]   2> 23367 T94 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty1
   [junit4]   2> 23367 T94 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty1/index/
   [junit4]   2> 23367 T94 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 23368 T94 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty1/index
   [junit4]   2> 23368 T94 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=17, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7989922721774966]
   [junit4]   2> 23369 T94 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2694cf lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@694c7),segFN=segments_1,generation=1}
   [junit4]   2> 23370 T94 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 23377 T94 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 23378 T94 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 23378 T94 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 23378 T94 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 23378 T94 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 23379 T94 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 23383 T94 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 23384 T94 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 23384 T94 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 23388 T94 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 23389 T94 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 23389 T94 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 23397 T94 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 23398 T94 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 23399 T94 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 23400 T94 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 23403 T94 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 23405 T94 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 23405 T94 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 23406 T94 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=9.20703125, floorSegmentMB=1.8662109375, forceMergeDeletesPctAllowed=5.930420212113585, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4004882394888162
   [junit4]   2> 23407 T94 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2694cf lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@694c7),segFN=segments_1,generation=1}
   [junit4]   2> 23407 T94 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 23408 T94 oass.SolrIndexSearcher.<init> Opening Searcher@892d63[collection1] main
   [junit4]   2> 23421 T95 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@892d63[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 23423 T94 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 23424 T48 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 23425 T48 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 23426 T98 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36156 collection:collection1 shard:shard2
   [junit4]   2> 23428 T98 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 23444 T98 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 23450 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23450 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23451 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23451 T98 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 23451 T98 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C41 name=collection1 org.apache.solr.core.SolrCore@15ff89f url=http://127.0.0.1:36156/collection1 node=127.0.0.1:36156_ C41_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:36156, core=collection1, node_name=127.0.0.1:36156_}
   [junit4]   2> 23451 T98 C41 P36156 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:36156/collection1/
   [junit4]   2> 23452 T98 C41 P36156 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 23452 T98 C41 P36156 oasc.SyncStrategy.syncToMe http://127.0.0.1:36156/collection1/ has no replicas
   [junit4]   2> 23452 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 23453 T98 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:36156/collection1/ shard2
   [junit4]   2> 23453 T98 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 23458 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23462 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23466 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23568 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 23568 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 23568 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 23613 T98 oasc.ZkController.register We are http://127.0.0.1:36156/collection1/ and leader is http://127.0.0.1:36156/collection1/
   [junit4]   2> 23613 T98 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36156
   [junit4]   2> 23613 T98 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 23613 T98 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 23613 T98 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 23615 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23615 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23615 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23615 T98 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 23617 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 23618 T72 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36156",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36156_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 23629 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 23731 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 23731 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 23731 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 23756 T48 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 23756 T48 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 23758 T48 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47455
   [junit4]   2> 23759 T48 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 23760 T48 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 23760 T48 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805
   [junit4]   2> 23760 T48 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/'
   [junit4]   2> 23770 T48 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/solr.xml
   [junit4]   2> 23784 T48 oasc.CoreContainer.<init> New CoreContainer 6099137
   [junit4]   2> 23785 T48 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/]
   [junit4]   2> 23785 T48 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 23786 T48 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 23786 T48 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 23786 T48 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 23787 T48 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 23787 T48 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 23788 T48 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 23788 T48 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 23788 T48 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 23791 T48 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 23791 T48 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 23791 T48 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 23791 T48 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42702/solr
   [junit4]   2> 23792 T48 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 23793 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 23795 T109 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11a85b4 name:ZooKeeperConnection Watcher:127.0.0.1:42702 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 23795 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 23797 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 23799 T111 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d698b name:ZooKeeperConnection Watcher:127.0.0.1:42702/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 23799 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 23822 T48 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 24826 T48 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47455_
   [junit4]   2> 24828 T48 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47455_
   [junit4]   2> 24831 T93 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 24831 T71 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 24832 T111 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 24832 T80 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 24842 T112 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 24842 T112 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 24845 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24845 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24845 T112 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 24846 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24847 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 24848 T72 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:47455",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47455_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 24848 T72 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 24849 T72 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 24851 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 24955 T111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24955 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24955 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 24955 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 25846 T112 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 25846 T112 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/collection1
   [junit4]   2> 25846 T112 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 25847 T112 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 25847 T112 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 25848 T112 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 25848 T112 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/collection1/'
   [junit4]   2> 25849 T112 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/collection1/lib/.svn/' to classloader
   [junit4]   2> 25849 T112 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/collection1/lib/classes/' to classloader
   [junit4]   2> 25849 T112 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/collection1/lib/README' to classloader
   [junit4]   2> 25876 T112 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 25895 T112 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 25896 T112 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 25903 T112 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 26045 T112 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 26047 T112 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 26048 T112 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 26051 T112 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 26075 T112 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 26075 T112 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ShardSplitTest-jetty2-1393614578805/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty2/
   [junit4]   2> 26075 T112 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@142865a
   [junit4]   2> 26076 T112 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty2
   [junit4]   2> 26077 T112 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty2/index/
   [junit4]   2> 26077 T112 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 26078 T112 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty2/index
   [junit4]   2> 26078 T112 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=17, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7989922721774966]
   [junit4]   2> 26079 T112 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@a4f837 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@27e451),segFN=segments_1,generation=1}
   [junit4]   2> 26079 T112 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 26086 T112 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 26087 T112 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 26087 T112 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 26087 T112 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 26088 T112 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 26088 T112 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 26088 T112 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 26089 T112 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 26089 T112 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 26090 T112 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 26091 T112 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 26091 T112 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 26091 T112 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 26092 T112 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 26093 T112 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 26094 T112 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 26096 T112 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 26098 T112 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 26098 T112 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 26100 T112 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=9.20703125, floorSegmentMB=1.8662109375, forceMergeDeletesPctAllowed=5.930420212113585, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4004882394888162
   [junit4]   2> 26101 T112 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@a4f837 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@27e451),segFN=segments_1,generation=1}
   [junit4]   2> 26101 T112 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 26102 T112 oass.SolrIndexSearcher.<init> Opening Searcher@fd4727[collection1] main
   [junit4]   2> 26108 T113 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@fd4727[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 26112 T112 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 26113 T48 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 26113 T48 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 26113 T116 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47455 collection:collection1 shard:shard1
   [junit4]   2> 26115 T116 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 26121 T116 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 26123 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26123 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26123 T116 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 26124 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26124 T116 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C42 name=collection1 org.apache.solr.core.SolrCore@12d4227 url=http://127.0.0.1:47455/collection1 node=127.0.0.1:47455_ C42_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:47455, core=collection1, node_name=127.0.0.1:47455_}
   [junit4]   2> 26124 T116 C42 P47455 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:47455/collection1/
   [junit4]   2> 26125 T116 C42 P47455 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 26125 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 26126 T116 C42 P47455 oasc.SyncStrategy.syncToMe http://127.0.0.1:47455/collection1/ has no replicas
   [junit4]   2> 26126 T116 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:47455/collection1/ shard1
   [junit4]   2> 26127 T116 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 26130 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26134 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26138 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26240 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 26240 T111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 26240 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 26240 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 26285 T116 oasc.ZkController.register We are http://127.0.0.1:47455/collection1/ and leader is http://127.0.0.1:47455/collection1/
   [junit4]   2> 26285 T116 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47455
   [junit4]   2> 26285 T116 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 26285 T116 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 26286 T116 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 26287 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26287 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26287 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26287 T116 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 26288 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 26289 T72 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:47455",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47455_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 26294 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 26339 T48 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 26340 T48 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 26342 T48 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51770
   [junit4]   2> 26343 T48 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 26344 T48 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 26344 T48 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494
   [junit4]   2> 26344 T48 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/'
   [junit4]   2> 26355 T48 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/solr.xml
   [junit4]   2> 26369 T48 oasc.CoreContainer.<init> New CoreContainer 26212847
   [junit4]   2> 26370 T48 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/]
   [junit4]   2> 26370 T48 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 26371 T48 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 26371 T48 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 26371 T48 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 26372 T48 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 26372 T48 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 26372 T48 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 26372 T48 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 26373 T48 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 26374 T48 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 26375 T48 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 26375 T48 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 26375 T48 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42702/solr
   [junit4]   2> 26376 T48 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 26376 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 26378 T127 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b263e3 name:ZooKeeperConnection Watcher:127.0.0.1:42702 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 26379 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 26382 T48 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 26383 T129 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5cf373 name:ZooKeeperConnection Watcher:127.0.0.1:42702/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 26383 T48 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 26387 T48 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 26397 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 26397 T129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 26397 T111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 26397 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 26397 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 27390 T48 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51770_
   [junit4]   2> 27392 T48 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51770_
   [junit4]   2> 27396 T111 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 27397 T80 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 27397 T129 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 27397 T93 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 27397 T71 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 27408 T130 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 27409 T130 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 27410 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27410 T130 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 27410 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27411 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27412 T72 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 27413 T72 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51770",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51770_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 27413 T72 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 27413 T72 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 27416 T71 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 27518 T71 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 27519 T93 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 27518 T111 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 27518 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 27518 T129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 28411 T130 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 28411 T130 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/collection1
   [junit4]   2> 28411 T130 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 28412 T130 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 28412 T130 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 28414 T130 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 28414 T130 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/collection1/'
   [junit4]   2> 28414 T130 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/collection1/lib/.svn/' to classloader
   [junit4]   2> 28415 T130 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/collection1/lib/classes/' to classloader
   [junit4]   2> 28415 T130 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/collection1/lib/README' to classloader
   [junit4]   2> 28449 T130 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
   [junit4]   2> 28464 T130 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 28465 T130 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 28470 T130 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 28623 T130 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 28625 T130 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 28625 T130 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 28628 T130 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 28650 T130 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 28650 T130 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-jetty3-1393614581494/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty3/
   [junit4]   2> 28650 T130 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@142865a
   [junit4]   2> 28651 T130 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty3
   [junit4]   2> 28652 T130 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty3/index/
   [junit4]   2> 28652 T130 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 28653 T130 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty3/index
   [junit4]   2> 28654 T130 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=17, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7989922721774966]
   [junit4]   2> 28655 T130 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@e14509 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1ab8da),segFN=segments_1,generation=1}
   [junit4]   2> 28656 T130 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 28661 T130 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 28662 T130 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 28662 T130 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 28662 T130 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 28662 T130 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 28663 T130 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 28663 T130 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 28663 T130 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 28664 T130 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 28664 T130 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 28665 T130 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 28665 T130 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]

[...truncated too long message...]

Elector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> ASYNC  NEW_CORE C52 name=collection1 org.apache.solr.core.SolrCore@1c0703b url=http://127.0.0.1:40234/collection1 node=127.0.0.1:40234_ C52_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:40234, core=collection1, node_name=127.0.0.1:40234_}
   [junit4]   2> 340333 T148 C52 P40234 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 340334 T148 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 340334 T148 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40234/collection1/ shard1
   [junit4]   2> 340334 T148 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1c0703b
   [junit4]   2> 340341 T148 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 340342 T148 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 340342 T148 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 340342 T148 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 340342 T148 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 340343 T148 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 340343 T148 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty4;done=false>>]
   [junit4]   2> 340343 T148 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty4
   [junit4]   2> 340343 T148 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty4/index;done=false>>]
   [junit4]   2> 340343 T148 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ShardSplitTest-1393614572244/jetty4/index
   [junit4]   2> 340343 T148 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 340344 T148 oasc.SolrException.log ERROR There was a problem trying to register as the leader:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:206)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:203)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:203)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:414)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:383)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:370)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 340344 T148 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election/91331924614250508-core_node4-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:171)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:171)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 340345 T148 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 340345 T148 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=DefaultSimilarity, locale=lv, timezone=SystemV/PST8PDT
   [junit4]   2> NOTE: Linux 3.8.0-36-generic i386/Oracle Corporation 1.8.0 (32-bit)/cpus=8,threads=1,free=12743496,total=64880640
   [junit4]   2> NOTE: All tests run in this JVM: [DocValuesTest, TestSurroundQueryParser, IndexSchemaTest, TestTrie, TestFunctionQuery, ReturnFieldsTest, TestMaxScoreQueryParser, ShardSplitTest]
   [junit4] Completed on J0 in 324.41s, 1 test, 1 error <<< FAILURES!

[...truncated 966 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:451: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1278: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:911: There were test failures: 363 suites, 1570 tests, 1 error, 36 ignored (8 assumptions)

Total time: 51 minutes 47 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-fcs-b129 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure