You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/03/11 01:41:16 UTC

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-fcs-b132) - Build # 9748 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9748/
Java: 64bit/jdk1.8.0-fcs-b132 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.client.solrj.impl.CloudSolrServerTest.testShutdown

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:34177 within 45000 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:34177 within 45000 ms
	at __randomizedtesting.SeedInfo.seed([454A32CC515D73A6:A63C3B59B627E6D4]:0)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:200)
	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:78)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:34177 within 45000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
	... 46 more




Build Log:
[...truncated 11583 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
   [junit4]   2> 12962 T52 oas.SolrTestCaseJ4.startTrackingSearchers WARN startTrackingSearchers: numOpens=7 numCloses=7
   [junit4]   2> 12963 T52 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 12964 T52 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /h_ebp/h
   [junit4]   2> 12967 T52 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
   [junit4]   2> 12972 T52 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1394498338970
   [junit4]   2> 12976 T52 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 12978 T53 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 13078 T52 oasc.ZkTestServer.run start zk server on port:34177
   [junit4]   2> 13125 T52 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 13136 T54 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 72427 T56 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 57483ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 72428 T52 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
   [junit4]   2> 72436 T54 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144ae9400e50000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 72437 T52 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:34177 34177
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrServerTest -Dtests.method=testShutdown -Dtests.seed=454A32CC515D73A6 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_AE -Dtests.timezone=America/Indianapolis -Dtests.file.encoding=UTF-8
   [junit4] ERROR   59.8s J0 | CloudSolrServerTest.testShutdown <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:34177 within 45000 ms
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([454A32CC515D73A6:A63C3B59B627E6D4]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:150)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:101)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:91)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:200)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:78)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:34177 within 45000 ms
   [junit4]    > 	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:223)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:142)
   [junit4]    > 	... 46 more
   [junit4]   2> 72803 T52 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1394498398800
   [junit4]   2> 72804 T52 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 72804 T60 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 72904 T52 oasc.ZkTestServer.run start zk server on port:50689
   [junit4]   2> 72905 T52 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 72958 T66 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6891d056 name:ZooKeeperConnection Watcher:127.0.0.1:50689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 72959 T52 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 72964 T52 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 72987 T52 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 72990 T68 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ccccc6b name:ZooKeeperConnection Watcher:127.0.0.1:50689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 72990 T52 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 72997 T52 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 73002 T52 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 73007 T52 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 73011 T52 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 73015 T52 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 73016 T52 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 73022 T52 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 73022 T52 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 73027 T52 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 73027 T52 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
   [junit4]   2> 73027 T52 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
   [junit4]   2> 73028 T52 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
   [junit4]   2> 73028 T52 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 73028 T52 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 73028 T52 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 73029 T52 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 73029 T52 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 73033 T52 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 73035 T70 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64de004e name:ZooKeeperConnection Watcher:127.0.0.1:50689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 73035 T52 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 73078 T52 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 73336 T52 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 73347 T52 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:53329
   [junit4]   2> 73352 T52 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 73352 T52 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 73353 T52 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1394498399035
   [junit4]   2> 73353 T52 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1394498399035/'
   [junit4]   2> 73367 T52 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1394498399035/solr.xml
   [junit4]   2> 73386 T52 oasc.CoreContainer.<init> New CoreContainer 2134399534
   [junit4]   2> 73387 T52 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1394498399035/]
   [junit4]   2> 73387 T52 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 73388 T52 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 73388 T52 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 73388 T52 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 73389 T52 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 73389 T52 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 73389 T52 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 73389 T52 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 73390 T52 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 73396 T52 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 73397 T52 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 73397 T52 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 73397 T52 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50689/solr
   [junit4]   2> 73405 T52 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 73406 T52 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 73410 T81 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57136a80 name:ZooKeeperConnection Watcher:127.0.0.1:50689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 73410 T52 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 73416 T52 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 73417 T83 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ac4a59b name:ZooKeeperConnection Watcher:127.0.0.1:50689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 73418 T52 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 73423 T52 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 73429 T52 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 73447 T52 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 73461 T52 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53329_h_ebp%2Fh
   [junit4]   2> 73463 T52 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53329_h_ebp%2Fh
   [junit4]   2> 73467 T52 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 73482 T52 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 73494 T52 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:53329_h_ebp%2Fh
   [junit4]   2> 73495 T52 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 73507 T52 oasc.Overseer.start Overseer (id=91389847064281092-127.0.0.1:53329_h_ebp%2Fh-n_0000000000) starting
   [junit4]   2> 73517 T52 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 73529 T85 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 73529 T52 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 73530 T85 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 73532 T52 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 73534 T52 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 73542 T84 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 73547 T86 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 73547 T86 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 73550 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 73551 T86 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 73568 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 73569 T84 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:53329/h_ebp/h",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53329_h_ebp%2Fh",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 73570 T84 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 73576 T84 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 73580 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 73585 T83 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 74551 T86 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 74551 T86 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1394498399035/collection1
   [junit4]   2> 74552 T86 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 74553 T86 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 74553 T86 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 74554 T86 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 74555 T86 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1394498399035/collection1/'
   [junit4]   2> 74581 T86 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 74597 T86 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 74599 T86 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 74625 T86 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 74641 T86 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableIntField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 74645 T86 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableLongField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 74649 T86 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableFloatField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 74653 T86 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableDoubleField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 74837 T86 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 74838 T86 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 74843 T86 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 74843 T86 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 74846 T86 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 74857 T86 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1394498399035/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/control/data/
   [junit4]   2> 74858 T86 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 74860 T86 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/control/data
   [junit4]   2> 74863 T86 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/control/data/index/
   [junit4]   2> 74863 T86 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 74865 T86 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/control/data/index
   [junit4]   2> 74868 T86 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@26512de5 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 74868 T86 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 74870 T86 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 74870 T86 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 74870 T86 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 74870 T86 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 74871 T86 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 74871 T86 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 74871 T86 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 74872 T86 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 74876 T86 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 74878 T86 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 74878 T86 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 74879 T86 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@26512de5 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 74883 T86 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 74883 T86 oass.SolrIndexSearcher.<init> Opening Searcher@4bb91cd[collection1] main
   [junit4]   2> 74885 T87 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4bb91cd[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 74885 T86 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 74886 T52 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 74886 T52 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 74886 T90 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:53329/h_ebp/h collection:control_collection shard:shard1
   [junit4]   2> 74914 T52 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 74915 T92 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57cb0a3 name:ZooKeeperConnection Watcher:127.0.0.1:50689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 74915 T90 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 74916 T52 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 74922 T52 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 74927 T90 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 74930 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 74931 T90 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 74931 T90 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C9 name=collection1 org.apache.solr.core.SolrCore@7ca325ae url=https://127.0.0.1:53329/h_ebp/h/collection1 node=127.0.0.1:53329_h_ebp%2Fh C9_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:53329/h_ebp/h, core=collection1, node_name=127.0.0.1:53329_h_ebp%2Fh}
   [junit4]   2> 74933 T90 C9 P53329 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:53329/h_ebp/h/collection1/
   [junit4]   2> 74934 T90 C9 P53329 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 74935 T90 C9 P53329 oasc.SyncStrategy.syncToMe https://127.0.0.1:53329/h_ebp/h/collection1/ has no replicas
   [junit4]   2> 74935 T90 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:53329/h_ebp/h/collection1/ shard1
   [junit4]   2> 74936 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 74938 T90 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 74943 T52 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 74945 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 74951 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 74988 T52 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 74990 T52 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 75009 T52 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 75018 T52 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:38255
   [junit4]   2> 75021 T52 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 75021 T52 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 75022 T52 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1394498400941
   [junit4]   2> 75022 T52 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1394498400941/'
   [junit4]   2> 75039 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75051 T52 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1394498400941/solr.xml
   [junit4]   2> 75064 T52 oasc.CoreContainer.<init> New CoreContainer 1189024140
   [junit4]   2> 75065 T52 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1394498400941/]
   [junit4]   2> 75065 T52 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 75066 T52 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 75066 T52 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 75066 T52 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 75067 T52 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 75067 T52 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 75068 T52 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 75068 T52 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 75068 T52 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 75074 T52 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 75075 T52 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 75075 T52 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 75076 T52 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50689/solr
   [junit4]   2> 75076 T52 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 75077 T52 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 75090 T103 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f137826 name:ZooKeeperConnection Watcher:127.0.0.1:50689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 75091 T52 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 75094 T52 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 75100 T105 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19d940ad name:ZooKeeperConnection Watcher:127.0.0.1:50689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 75100 T52 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 75107 T52 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 75142 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 75143 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 75142 T83 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 75203 T90 oasc.ZkController.register We are https://127.0.0.1:53329/h_ebp/h/collection1/ and leader is https://127.0.0.1:53329/h_ebp/h/collection1/
   [junit4]   2> 75204 T90 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:53329/h_ebp/h
   [junit4]   2> 75204 T90 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 75204 T90 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 75204 T90 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 75205 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75205 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75205 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75206 T90 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 75208 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 75209 T84 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:53329/h_ebp/h",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53329_h_ebp%2Fh",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 75211 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75314 T83 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 75314 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 75314 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 76111 T52 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38255_h_ebp%2Fh
   [junit4]   2> 76113 T52 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38255_h_ebp%2Fh
   [junit4]   2> 76117 T105 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 76117 T92 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 76118 T83 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 76127 T106 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 76127 T106 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 76129 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76129 T106 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 76129 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76129 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76131 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 76131 T84 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:38255/h_ebp/h",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38255_h_ebp%2Fh",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 76132 T84 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 76132 T84 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 76134 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76240 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 76240 T83 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 76240 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 77129 T106 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 77129 T106 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1394498400941/collection1
   [junit4]   2> 77129 T106 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 77130 T106 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 77130 T106 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 77132 T106 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 77132 T106 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1394498400941/collection1/'
   [junit4]   2> 77146 T106 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 77153 T106 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 77155 T106 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 77167 T106 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 77280 T106 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 77281 T106 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 77283 T106 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 77283 T106 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 77284 T106 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 77284 T106 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1394498400941/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty1/
   [junit4]   2> 77285 T106 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 77285 T106 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty1
   [junit4]   2> 77286 T106 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty1/index/
   [junit4]   2> 77286 T106 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 77286 T106 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty1/index
   [junit4]   2> 77288 T106 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@54710666 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 77289 T106 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 77290 T106 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 77290 T106 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 77290 T106 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 77290 T106 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 77290 T106 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 77291 T106 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 77291 T106 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 77291 T106 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 77292 T106 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 77294 T106 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 77294 T106 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 77295 T106 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@54710666 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 77295 T106 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 77296 T106 oass.SolrIndexSearcher.<init> Opening Searcher@351d3548[collection1] main
   [junit4]   2> 77297 T106 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 77297 T107 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@351d3548[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 77299 T52 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 77299 T52 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 77302 T110 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:38255/h_ebp/h collection:collection1 shard:shard2
   [junit4]   2> 77304 T110 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 77315 T110 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 77328 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77328 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77328 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77329 T110 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 77329 T110 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C10 name=collection1 org.apache.solr.core.SolrCore@a3a3f9e url=https://127.0.0.1:38255/h_ebp/h/collection1 node=127.0.0.1:38255_h_ebp%2Fh C10_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:38255/h_ebp/h, core=collection1, node_name=127.0.0.1:38255_h_ebp%2Fh}
   [junit4]   2> 77330 T110 C10 P38255 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:38255/h_ebp/h/collection1/
   [junit4]   2> 77330 T110 C10 P38255 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 77330 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 77330 T110 C10 P38255 oasc.SyncStrategy.syncToMe https://127.0.0.1:38255/h_ebp/h/collection1/ has no replicas
   [junit4]   2> 77331 T110 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:38255/h_ebp/h/collection1/ shard2
   [junit4]   2> 77331 T110 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 77337 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77341 T52 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 77342 T52 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 77342 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77370 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77373 T52 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 77375 T52 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:36789
   [junit4]   2> 77377 T52 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 77377 T52 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 77378 T52 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1394498403298
   [junit4]   2> 77378 T52 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1394498403298/'
   [junit4]   2> 77399 T52 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1394498403298/solr.xml
   [junit4]   2> 77409 T52 oasc.CoreContainer.<init> New CoreContainer 397775563
   [junit4]   2> 77409 T52 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1394498403298/]
   [junit4]   2> 77409 T52 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 77410 T52 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 77410 T52 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 77410 T52 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 77410 T52 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 77411 T52 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 77411 T52 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 77411 T52 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 77411 T52 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 77416 T52 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 77417 T52 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 77417 T52 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 77418 T52 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50689/solr
   [junit4]   2> 77418 T52 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 77418 T52 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 77424 T121 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1521fee4 name:ZooKeeperConnection Watcher:127.0.0.1:50689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 77424 T52 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 77429 T52 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 77431 T123 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4757f40b name:ZooKeeperConnection Watcher:127.0.0.1:50689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 77431 T52 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 77436 T52 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 77474 T83 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 77474 T123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 77474 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 77474 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 77492 T110 oasc.ZkController.register We are https://127.0.0.1:38255/h_ebp/h/collection1/ and leader is https://127.0.0.1:38255/h_ebp/h/collection1/
   [junit4]   2> 77493 T110 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:38255/h_ebp/h
   [junit4]   2> 77493 T110 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 77493 T110 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 77493 T110 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 77494 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77495 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77495 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77495 T110 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 77496 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 77497 T84 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:38255/h_ebp/h",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38255_h_ebp%2Fh",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 77500 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 77603 T83 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 77603 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 77606 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 77607 T123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 78439 T52 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36789_h_ebp%2Fh
   [junit4]   2> 78441 T52 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36789_h_ebp%2Fh
   [junit4]   2> 78444 T83 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 78444 T123 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 78444 T92 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 78445 T105 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 78459 T124 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 78459 T124 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 78461 T124 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 78461 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78462 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78462 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78464 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 78465 T84 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:36789/h_ebp/h",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36789_h_ebp%2Fh",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 78465 T84 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 78465 T84 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 78469 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78573 T123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 78574 T83 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 78573 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 78573 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 79461 T124 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 79462 T124 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1394498403298/collection1
   [junit4]   2> 79462 T124 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 79463 T124 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 79463 T124 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 79471 T124 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 79472 T124 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1394498403298/collection1/'
   [junit4]   2> 79495 T124 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 79503 T124 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 79505 T124 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 79530 T124 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 79668 T124 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 79670 T124 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 79672 T124 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 79673 T124 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 79674 T124 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 79674 T124 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1394498403298/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty2/
   [junit4]   2> 79675 T124 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 79676 T124 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty2
   [junit4]   2> 79677 T124 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty2/index/
   [junit4]   2> 79677 T124 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 79677 T124 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty2/index
   [junit4]   2> 79679 T124 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@318c6e0b lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 79680 T124 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 79681 T124 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 79681 T124 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 79681 T124 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 79681 T124 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 79681 T124 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 79682 T124 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 79682 T124 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 79682 T124 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 79683 T124 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 79684 T124 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 79685 T124 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 79686 T124 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@318c6e0b lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 79686 T124 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 79686 T124 oass.SolrIndexSearcher.<init> Opening Searcher@2e6f39d5[collection1] main
   [junit4]   2> 79687 T125 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2e6f39d5[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 79687 T124 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 79688 T52 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 79689 T52 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 79689 T128 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:36789/h_ebp/h collection:collection1 shard:shard1
   [junit4]   2> 79690 T128 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 79704 T128 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 79707 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79707 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79708 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79709 T128 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 79709 T128 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 79709 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C11 name=collection1 org.apache.solr.core.SolrCore@1d312faa url=https://127.0.0.1:36789/h_ebp/h/collection1 node=127.0.0.1:36789_h_ebp%2Fh C11_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:36789/h_ebp/h, core=collection1, node_name=127.0.0.1:36789_h_ebp%2Fh}
   [junit4]   2> 79709 T128 C11 P36789 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:36789/h_ebp/h/collection1/
   [junit4]   2> 79713 T128 C11 P36789 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 79714 T128 C11 P36789 oasc.SyncStrategy.syncToMe https://127.0.0.1:36789/h_ebp/h/collection1/ has no replicas
   [junit4]   2> 79714 T128 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:36789/h_ebp/h/collection1/ shard1
   [junit4]   2> 79715 T128 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 79715 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79724 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79732 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79743 T52 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 79747 T52 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 79807 T52 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 79820 T52 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50671
   [junit4]   2> 79823 T52 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 79823 T52 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 79824 T52 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1394498405692
   [junit4]   2> 79824 T52 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1394498405692/'
   [junit4]   2> 79836 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 79836 T123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 79837 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 79837 T83 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 79856 T52 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1394498405692/solr.xml
   [junit4]   2> 79874 T128 oasc.ZkController.register We are https://127.0.0.1:36789/h_ebp/h/collection1/ and leader is https://127.0.0.1:36789/h_ebp/h/collection1/
   [junit4]   2> 79875 T128 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:36789/h_ebp/h
   [junit4]   2> 79875 T128 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 79875 T128 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 79875 T128 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 79876 T52 oasc.CoreContainer.<init> New CoreContainer 408522205
   [junit4]   2> 79876 T52 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1394498405692/]
   [junit4]   2> 79877 T52 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 79877 T52 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 79877 T52 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 79877 T52 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 79878 T52 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 79878 T52 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 79878 T52 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 79878 T52 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 79879 T52 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 79882 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79882 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79882 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79883 T128 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 79885 T52 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 79886 T52 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 79886 T52 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 79886 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 79893 T52 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50689/solr
   [junit4]   2> 79893 T52 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 79893 T84 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:36789/h_ebp/h",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36789_h_ebp%2Fh",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 79905 T52 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 79906 T139 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@290a0f33 name:ZooKeeperConnection Watcher:127.0.0.1:50689 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 79906 T52 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 79908 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 79914 T52 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 79922 T141 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2479c606 name:ZooKeeperConnection Watcher:127.0.0.1:50689/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 79923 T52 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 79931 T52 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 80016 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 80017 T123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 80017 T83 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 80017 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 80017 T141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 80935 T52 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50671_h_ebp%2Fh
   [junit4]   2> 80937 T52 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50671_h_ebp%2Fh
   [junit4]   2> 80940 T123 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 80940 T92 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 80940 T105 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 80941 T83 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 80940 T141 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 80948 T142 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 80949 T142 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 80950 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 80950 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 80950 T142 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 80950 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 80952 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 80953 T84 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50671/h_ebp/h",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50671_h_ebp%2Fh",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 80953 T84 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 80953 T84 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 80958 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 81060 T123 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 81060 T83 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 81060 T105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 81061 T141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 81060 T92 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 81951 T142 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 81951 T142 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1394498405692/collection1
   [junit4]   2> 81952 T142 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 81953 T142 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 81953 T142 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 81955 T142 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 81955 T142 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1394498405692/collection1/'
   [junit4]   2> 81986 T142 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 81992 T142 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 81994 T142 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 82003 T142 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 82104 T142 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 82105 T142 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 82106 T142 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 82106 T142 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 82107 T142 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 82107 T142 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1394498405692/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty3/
   [junit4]   2> 82107 T142 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 82108 T142 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty3
   [junit4]   2> 82108 T142 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty3/index/
   [junit4]   2> 82108 T142 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 82108 T142 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty3/index
   [junit4]   2> 82109 T142 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@61fe027f lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty3/index),segFN=segments_1,generation=1}
   [junit4]   2> 82110 T142 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 82110 T142 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 82111 T142 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 82111 T142 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 82111 T142 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 82111 T142 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 82111 T142 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 82111 T142 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 82112 T142 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 82113 T142 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 82114 T142 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 82114 T142 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 82115 T142 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@61fe027f lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty3/index),segFN=segments_1,generation=1}
   [junit4]   2> 82115 T142 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 82116 T142 oass.SolrIndexSearcher.<init> Opening Searcher@bc36ad1[collection1] main
   [junit4]   2> 82116 T143 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@bc36ad1[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 82116 T142 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 82117 T52 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 82117 T52 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 82117 T146 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50671/h_ebp/h collection:collection1 shard:shard2
   [junit4]   2> 82122 T146 oasc.ZkController.register We are https://127.0.0.1:50671/h_ebp/h/collection1/ and leader is https://127.0.0.1:38255/h_ebp/h/collection1/
   [junit4]   2> 82122 T146 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50671/h_ebp/h
   [junit4]   2> 82122 T146 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 82122 T146 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C12 name=collection1 org.apache.solr.core.SolrCore@1395507b url=https://127.0.0.1:50671/h_ebp/h/collection1 node=127.0.0.1:50671_h_ebp%2Fh C12_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:50671/h_ebp/h, core=collection1, node_name=127.0.0.1:50671_h_ebp%2Fh}
   [junit4]   2> 82126 T147 C12 P50671 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 82126 T146 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 82126 T147 C12 P50671 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 82127 T147 C12 P50671 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
   [junit4]   2> 82127 T147 C12 P50671 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 82128 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 82128 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 82128 T83 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 82129 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 82130 T84 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50671/h_ebp/h",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50671_h_ebp%2Fh",
   [junit4]   2> 	  "shard":"shard2",
   [junit4

[...truncated too long message...]

seJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 92844 T52 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50689 50689
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 93160 T52 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 93872 T83 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 94087 T123 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 94990 T141 C21 P50671 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 94998 T141 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 94999 T141 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1395507b
   [junit4]   2> 95000 T141 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=1,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=5}
   [junit4]   2> 95000 T141 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 95000 T141 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 95001 T141 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 95003 T141 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 95003 T141 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 95003 T141 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty3 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty3;done=false>>]
   [junit4]   2> 95004 T141 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty3
   [junit4]   2> 95004 T141 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty3/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty3/index;done=false>>]
   [junit4]   2> 95004 T141 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty3/index
   [junit4]   2> 95005 T141 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 95005 T141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 95005 T141 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 95006 T141 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 95093 T160 C24 P44255 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 95094 T160 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 95094 T160 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@192993b4
   [junit4]   2> 95095 T160 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=1,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=5}
   [junit4]   2> 95095 T160 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 95095 T160 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 95095 T160 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 95097 T160 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 95097 T160 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 95097 T160 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty4;done=false>>]
   [junit4]   2> 95097 T160 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty4
   [junit4]   2> 95097 T160 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty4/index;done=false>>]
   [junit4]   2> 95097 T160 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1394498398800/jetty4/index
   [junit4]   2> 95097 T160 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:259)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 95098 T160 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=ar_AE, timezone=America/Indianapolis
   [junit4]   2> NOTE: Linux 3.8.0-36-generic amd64/Oracle Corporation 1.8.0 (64-bit)/cpus=8,threads=1,free=107259864,total=189792256
   [junit4]   2> NOTE: All tests run in this JVM: [AnlysisResponseBaseTest, SolrExampleXMLTest, TestCoreAdmin, CloudSolrServerTest]
   [junit4] Completed on J0 in 82.23s, 2 tests, 1 error <<< FAILURES!

[...truncated 130 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1274: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 49 suites, 283 tests, 1 error

Total time: 46 minutes 23 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-fcs-b132 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure