You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/02/23 03:47:15 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9565 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9565/
Java: 32bit/jdk1.7.0_51 -server -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.client.solrj.impl.CloudSolrServerTest.testShutdown

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:42527 within 30000 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:42527 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([668FC3D7B747DFE9:85F9CA42503D4A9B]:0)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:148)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:94)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:85)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:200)
	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:77)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:42527 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:222)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:140)
	... 47 more




Build Log:
[...truncated 11542 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
   [junit4]   2> 8578 T50 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 8579 T50 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 8581 T50 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
   [junit4]   2> 8588 T50 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1393123495502
   [junit4]   2> 8591 T50 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 8593 T51 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 8765 T50 oasc.ZkTestServer.run start zk server on port:42527
   [junit4]   2> 8819 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 62071 T54 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 53231ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 62073 T50 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
   [junit4]   2> 62085 T52 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1445ca18e910000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 62087 T50 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:42527 42527
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrServerTest -Dtests.method=testShutdown -Dtests.seed=668FC3D7B747DFE9 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_CA -Dtests.timezone=Asia/Kuwait -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   58.7s J0 | CloudSolrServerTest.testShutdown <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:42527 within 30000 ms
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([668FC3D7B747DFE9:85F9CA42503D4A9B]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:148)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:94)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:85)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:200)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:77)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:42527 within 30000 ms
   [junit4]    > 	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:222)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:140)
   [junit4]    > 	... 47 more
   [junit4]   2> 67247 T50 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1393123554161
   [junit4]   2> 67248 T50 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 67248 T58 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 67349 T50 oasc.ZkTestServer.run start zk server on port:50990
   [junit4]   2> 67350 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 67355 T64 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e5f22d name:ZooKeeperConnection Watcher:127.0.0.1:50990 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 67355 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 67357 T50 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 67386 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 67388 T66 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@134b9f2 name:ZooKeeperConnection Watcher:127.0.0.1:50990/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 67388 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 67393 T50 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 67398 T50 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 67403 T50 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 67406 T50 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 67415 T50 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 67423 T50 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 67431 T50 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 67432 T50 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 67436 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 67437 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
   [junit4]   2> 67437 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
   [junit4]   2> 67438 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
   [junit4]   2> 67438 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 67439 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 67439 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 67440 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 67440 T50 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 67446 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 67448 T68 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e3f1e4 name:ZooKeeperConnection Watcher:127.0.0.1:50990/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 67449 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 67493 T50 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 67691 T50 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 67695 T50 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:58667
   [junit4]   2> 67702 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 67702 T50 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 67703 T50 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393123554365
   [junit4]   2> 67703 T50 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393123554365/'
   [junit4]   2> 67716 T50 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393123554365/solr.xml
   [junit4]   2> 67797 T50 oasc.CoreContainer.<init> New CoreContainer 13061785
   [junit4]   2> 67798 T50 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393123554365/]
   [junit4]   2> 67800 T50 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 67801 T50 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 67801 T50 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 67802 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 67802 T50 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 67802 T50 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 67803 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 67803 T50 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 67803 T50 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 67810 T50 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 67811 T50 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 67811 T50 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 67812 T50 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50990/solr
   [junit4]   2> 67823 T50 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 67825 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 67828 T79 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1450cac name:ZooKeeperConnection Watcher:127.0.0.1:50990 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 67828 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 67834 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 67836 T81 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f7cd49 name:ZooKeeperConnection Watcher:127.0.0.1:50990/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 67836 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 67841 T50 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 67850 T50 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 67872 T50 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 67875 T50 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58667_
   [junit4]   2> 67878 T50 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58667_
   [junit4]   2> 67884 T50 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 67888 T50 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 67899 T50 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:58667_
   [junit4]   2> 67900 T50 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 67903 T50 oasc.Overseer.start Overseer (id=91299745245691908-127.0.0.1:58667_-n_0000000000) starting
   [junit4]   2> 67915 T50 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 67929 T50 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 67929 T83 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 67931 T83 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 67933 T50 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 67936 T50 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 67946 T82 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 67958 T84 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 67958 T84 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 67962 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 67962 T84 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 67965 T82 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 67966 T82 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58667",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:58667_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 67966 T82 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 67973 T82 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 67978 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 67983 T81 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 68963 T84 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 68963 T84 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393123554365/collection1
   [junit4]   2> 68964 T84 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 68966 T84 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 68966 T84 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 68970 T84 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 68971 T84 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393123554365/collection1/'
   [junit4]   2> 69033 T84 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 69090 T84 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 69092 T84 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 69175 T84 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 69266 T84 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableIntField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 69274 T84 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableLongField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 69282 T84 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableFloatField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 69290 T84 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableDoubleField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 69702 T84 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 69705 T84 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 69708 T84 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 69708 T84 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 69711 T84 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 69712 T84 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1393123554365/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/control/data/
   [junit4]   2> 69712 T84 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 69713 T84 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/control/data
   [junit4]   2> 69713 T84 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/control/data/index/
   [junit4]   2> 69713 T84 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 69714 T84 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/control/data/index
   [junit4]   2> 69715 T84 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@cdb594 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 69715 T84 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 69716 T84 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 69717 T84 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 69717 T84 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 69717 T84 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 69721 T84 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 69721 T84 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 69722 T84 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 69722 T84 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 69724 T84 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 69727 T84 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 69728 T84 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 69728 T84 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@cdb594 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/control/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 69729 T84 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 69729 T84 oass.SolrIndexSearcher.<init> Opening Searcher@f62267[collection1] main
   [junit4]   2> 69730 T85 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f62267[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 69730 T84 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 69732 T50 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 69732 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 69732 T88 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:58667 collection:control_collection shard:shard1
   [junit4]   2> 69739 T88 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 69741 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 69744 T90 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a809b9 name:ZooKeeperConnection Watcher:127.0.0.1:50990/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 69744 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 69746 T50 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 69752 T88 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 69753 T50 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 69756 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 69757 T88 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 69758 T88 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 69759 T82 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C6 name=collection1 org.apache.solr.core.SolrCore@1ea1ada url=https://127.0.0.1:58667/collection1 node=127.0.0.1:58667_ C6_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:58667, core=collection1, node_name=127.0.0.1:58667_}
   [junit4]   2> 69760 T88 C6 P58667 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:58667/collection1/
   [junit4]   2> 69760 T88 C6 P58667 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 69761 T88 C6 P58667 oasc.SyncStrategy.syncToMe https://127.0.0.1:58667/collection1/ has no replicas
   [junit4]   2> 69761 T88 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:58667/collection1/ shard1
   [junit4]   2> 69761 T88 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 69766 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 69773 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 69774 T81 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 69783 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 69787 T82 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 69792 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 69794 T50 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 69795 T50 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 69801 T50 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 69803 T50 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:44601
   [junit4]   2> 69806 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 69807 T50 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 69807 T50 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393123556667
   [junit4]   2> 69807 T50 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393123556667/'
   [junit4]   2> 69823 T50 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393123556667/solr.xml
   [junit4]   2> 69874 T50 oasc.CoreContainer.<init> New CoreContainer 16165400
   [junit4]   2> 69875 T50 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393123556667/]
   [junit4]   2> 69876 T50 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 69876 T50 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 69877 T50 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 69877 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 69877 T50 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 69878 T50 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 69878 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 69878 T50 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 69879 T50 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 69885 T50 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 69886 T50 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 69886 T50 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 69887 T50 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50990/solr
   [junit4]   2> 69887 T50 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 69888 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 69892 T101 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d72085 name:ZooKeeperConnection Watcher:127.0.0.1:50990 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 69892 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 69895 T81 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 69895 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 69897 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 69900 T103 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17dd3de name:ZooKeeperConnection Watcher:127.0.0.1:50990/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 69900 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 69908 T50 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 69935 T88 oasc.ZkController.register We are https://127.0.0.1:58667/collection1/ and leader is https://127.0.0.1:58667/collection1/
   [junit4]   2> 69936 T88 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:58667
   [junit4]   2> 69936 T88 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 69936 T88 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 69936 T88 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 69938 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 69938 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 69938 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 69939 T88 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 69941 T82 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 69942 T82 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58667",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:58667_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 69946 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 70049 T81 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 70049 T103 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 70049 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 70912 T50 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44601_
   [junit4]   2> 70915 T50 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44601_
   [junit4]   2> 70920 T81 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 70920 T103 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 70920 T90 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 70933 T104 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 70933 T104 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 70935 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 70935 T104 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 70936 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 70936 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 70938 T82 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 70940 T82 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44601",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:44601_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 70940 T82 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 70941 T82 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 70945 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 71049 T103 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 71050 T81 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 71049 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 71936 T104 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 71936 T104 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393123556667/collection1
   [junit4]   2> 71936 T104 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 71938 T104 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 71938 T104 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 71940 T104 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 71940 T104 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393123556667/collection1/'
   [junit4]   2> 71983 T104 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 72027 T104 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 72033 T104 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 72084 T104 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 72477 T104 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 72483 T104 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 72488 T104 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 72488 T104 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 72490 T104 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 72490 T104 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1393123556667/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty1/
   [junit4]   2> 72491 T104 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 72492 T104 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty1
   [junit4]   2> 72492 T104 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty1/index/
   [junit4]   2> 72492 T104 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 72493 T104 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty1/index
   [junit4]   2> 72495 T104 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1f68f29 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 72495 T104 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 72497 T104 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 72498 T104 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 72498 T104 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 72498 T104 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 72499 T104 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 72499 T104 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 72499 T104 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 72500 T104 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 72502 T104 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 72505 T104 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 72506 T104 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 72507 T104 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1f68f29 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty1/index),segFN=segments_1,generation=1}
   [junit4]   2> 72507 T104 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 72507 T104 oass.SolrIndexSearcher.<init> Opening Searcher@134af37[collection1] main
   [junit4]   2> 72508 T105 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@134af37[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 72508 T104 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 72510 T50 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 72510 T108 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:44601 collection:collection1 shard:shard2
   [junit4]   2> 72510 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 72512 T108 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 72527 T108 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 72532 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 72533 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 72533 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 72535 T108 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 72535 T108 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C7 name=collection1 org.apache.solr.core.SolrCore@3b0990 url=https://127.0.0.1:44601/collection1 node=127.0.0.1:44601_ C7_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:44601, core=collection1, node_name=127.0.0.1:44601_}
   [junit4]   2> 72535 T108 C7 P44601 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:44601/collection1/
   [junit4]   2> 72536 T108 C7 P44601 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 72537 T82 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 72537 T108 C7 P44601 oasc.SyncStrategy.syncToMe https://127.0.0.1:44601/collection1/ has no replicas
   [junit4]   2> 72538 T108 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:44601/collection1/ shard2
   [junit4]   2> 72538 T108 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 72542 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 72543 T50 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 72544 T50 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 72548 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 72549 T50 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 72554 T50 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:49928
   [junit4]   2> 72557 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 72558 T50 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 72558 T50 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393123559426
   [junit4]   2> 72559 T50 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393123559426/'
   [junit4]   2> 72562 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 72565 T81 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 72565 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 72566 T103 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 72580 T50 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393123559426/solr.xml
   [junit4]   2> 72600 T108 oasc.ZkController.register We are https://127.0.0.1:44601/collection1/ and leader is https://127.0.0.1:44601/collection1/
   [junit4]   2> 72600 T108 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:44601
   [junit4]   2> 72601 T108 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 72601 T108 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 72601 T108 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 72603 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 72604 T108 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 72608 T82 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 72610 T82 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44601",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:44601_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 72617 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 72646 T50 oasc.CoreContainer.<init> New CoreContainer 29012031
   [junit4]   2> 72647 T50 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393123559426/]
   [junit4]   2> 72649 T50 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 72650 T50 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 72650 T50 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 72651 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 72651 T50 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 72652 T50 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 72652 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 72653 T50 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 72655 T50 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 72666 T50 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 72667 T50 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 72668 T50 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 72668 T50 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50990/solr
   [junit4]   2> 72668 T50 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 72669 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 72671 T119 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fcedd name:ZooKeeperConnection Watcher:127.0.0.1:50990 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 72672 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 72680 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 72683 T121 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e8844a name:ZooKeeperConnection Watcher:127.0.0.1:50990/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 72683 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 72694 T50 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 72720 T103 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 72720 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 72720 T121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 72720 T81 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 73700 T50 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49928_
   [junit4]   2> 73702 T50 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49928_
   [junit4]   2> 73705 T81 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 73705 T103 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 73706 T90 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 73706 T121 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 73720 T122 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 73721 T122 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 73722 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 73722 T122 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 73722 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 73723 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 73725 T82 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 73727 T82 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:49928",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49928_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 73727 T82 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 73727 T82 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 73731 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 73834 T103 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 73834 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 73834 T121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 73834 T81 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 74723 T122 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 74723 T122 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393123559426/collection1
   [junit4]   2> 74724 T122 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 74725 T122 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 74725 T122 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 74727 T122 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 74727 T122 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393123559426/collection1/'
   [junit4]   2> 74770 T122 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 74818 T122 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 74820 T122 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 74871 T122 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 75250 T122 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 75253 T122 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 75256 T122 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 75256 T122 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 75257 T122 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 75258 T122 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1393123559426/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty2/
   [junit4]   2> 75258 T122 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 75259 T122 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty2
   [junit4]   2> 75259 T122 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty2/index/
   [junit4]   2> 75259 T122 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 75259 T122 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty2/index
   [junit4]   2> 75261 T122 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1321163 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 75261 T122 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 75262 T122 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 75262 T122 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 75263 T122 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 75263 T122 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 75263 T122 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 75264 T122 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 75264 T122 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 75264 T122 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 75266 T122 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 75269 T122 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 75270 T122 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 75270 T122 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1321163 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty2/index),segFN=segments_1,generation=1}
   [junit4]   2> 75271 T122 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 75271 T122 oass.SolrIndexSearcher.<init> Opening Searcher@12ebbac[collection1] main
   [junit4]   2> 75272 T123 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@12ebbac[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 75272 T122 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 75273 T50 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 75274 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 75273 T126 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:49928 collection:collection1 shard:shard1
   [junit4]   2> 75276 T126 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 75286 T126 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 75294 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75295 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75295 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75296 T126 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 75296 T126 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C8 name=collection1 org.apache.solr.core.SolrCore@8887c2 url=https://127.0.0.1:49928/collection1 node=127.0.0.1:49928_ C8_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:49928, core=collection1, node_name=127.0.0.1:49928_}
   [junit4]   2> 75296 T126 C8 P49928 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:49928/collection1/
   [junit4]   2> 75297 T126 C8 P49928 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 75297 T82 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 75297 T126 C8 P49928 oasc.SyncStrategy.syncToMe https://127.0.0.1:49928/collection1/ has no replicas
   [junit4]   2> 75298 T126 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:49928/collection1/ shard1
   [junit4]   2> 75298 T126 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 75303 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75307 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75310 T50 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 75311 T50 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 75312 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75318 T50 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 75322 T50 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:36356
   [junit4]   2> 75325 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 75325 T50 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 75326 T50 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393123562189
   [junit4]   2> 75326 T50 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393123562189/'
   [junit4]   2> 75338 T50 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393123562189/solr.xml
   [junit4]   2> 75389 T50 oasc.CoreContainer.<init> New CoreContainer 23963231
   [junit4]   2> 75390 T50 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393123562189/]
   [junit4]   2> 75391 T50 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 75392 T50 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 75392 T50 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 75393 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 75393 T50 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 75393 T50 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 75394 T50 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 75394 T50 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 75394 T50 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 75400 T50 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 75401 T50 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 75402 T50 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 75402 T50 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50990/solr
   [junit4]   2> 75402 T50 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 75403 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 75405 T137 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d14f1e name:ZooKeeperConnection Watcher:127.0.0.1:50990 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 75406 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 75408 T50 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 75410 T139 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f93c7 name:ZooKeeperConnection Watcher:127.0.0.1:50990/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 75411 T50 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 75417 T121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 75417 T81 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 75417 T103 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 75417 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 75421 T50 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 75458 T126 oasc.ZkController.register We are https://127.0.0.1:49928/collection1/ and leader is https://127.0.0.1:49928/collection1/
   [junit4]   2> 75458 T126 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:49928
   [junit4]   2> 75459 T126 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 75459 T126 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 75459 T126 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 75461 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75462 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75462 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75462 T126 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 75467 T82 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 75469 T82 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:49928",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49928_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 75473 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 75576 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 75576 T139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 75576 T81 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 75576 T121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 75576 T103 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 76427 T50 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36356_
   [junit4]   2> 76429 T50 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36356_
   [junit4]   2> 76433 T121 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 76433 T103 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 76433 T139 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 76433 T90 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 76434 T81 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 76442 T140 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 76442 T140 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 76443 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76444 T140 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 76444 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76444 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76446 T82 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 76447 T82 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:36356",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36356_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 76448 T82 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 76448 T82 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 76452 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 76556 T139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 76556 T81 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 76556 T90 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 76556 T103 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 76556 T121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 77444 T140 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 77445 T140 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393123562189/collection1
   [junit4]   2> 77445 T140 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 77446 T140 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 77446 T140 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 77448 T140 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 77449 T140 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393123562189/collection1/'
   [junit4]   2> 77503 T140 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 77560 T140 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 77562 T140 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 77611 T140 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 78026 T140 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 78029 T140 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 78031 T140 oass.IndexSchema.readSchema WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 78031 T140 oass.IndexSchema.readSchema WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 78032 T140 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 78033 T140 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1393123562189/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty3/
   [junit4]   2> 78033 T140 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 78034 T140 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty3
   [junit4]   2> 78034 T140 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty3/index/
   [junit4]   2> 78034 T140 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 78034 T140 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty3/index
   [junit4]   2> 78036 T140 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@349074 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty3/index),segFN=segments_1,generation=1}
   [junit4]   2> 78036 T140 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 78037 T140 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 78037 T140 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 78038 T140 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 78038 T140 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 78038 T140 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 78038 T140 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 78039 T140 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 78039 T140 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 78041 T140 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 78044 T140 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 78044 T140 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 78045 T140 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@349074 lockFactory=NativeFSLockFactory@./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty3/index),segFN=segments_1,generation=1}
   [junit4]   2> 78045 T140 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 78046 T140 oass.SolrIndexSearcher.<init> Opening Searcher@23922a[collection1] main
   [junit4]   2> 78047 T141 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@23922a[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 78047 T140 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 78048 T50 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 78048 T50 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 78048 T144 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:36356 collection:collection1 shard:shard2
   [junit4]   2> 78053 T144 oasc.ZkController.register We are https://127.0.0.1:36356/collection1/ and leader is https://127.0.0.1:44601/collection1/
   [junit4]   2> 78054 T144 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:36356
   [junit4]   2> 78054 T144 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 78054 T144 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C9 name=collection1 org.apache.solr.core.SolrCore@815231 url=https://127.0.0.1:36356/collection1 node=127.0.0.1:36356_ C9_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:36356, core=collection1, node_name=127.0.0.1:36356_}
   [junit4]   2> 78057 T145 C9 P36356 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 78058 T145 C9 P36356 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 78058 T144 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 78058 T145 C9 P36356 oasc.ZkController.publish publishing core=collection1 state=recovering
   [junit4]   2> 78059 T145 C9 P36356 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 78061 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78061 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78061 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78064 T82 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 78066 T82 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:36356",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:36356_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node3"}
   [junit4]   2> 78072 T81 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 78075 T121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/c

[...truncated too long message...]

7 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> ASYNC  NEW_CORE C34 name=collection1 org.apache.solr.core.SolrCore@815231 url=https://127.0.0.1:36356/collection1 node=127.0.0.1:36356_ C34_STATE=coll:collection1 core:collection1 props:{state=active, base_url=https://127.0.0.1:36356, core=collection1, node_name=127.0.0.1:36356_}
   [junit4]   2> 91769 T139 C34 P36356 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 90850 T121 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 99641 T50 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 99643 T139 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 99643 T158 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 99644 T139 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@815231
   [junit4]   2> 99644 T158 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@12a0ff4
   [junit4]   2> 99645 T139 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=1,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=5}
   [junit4]   2> 99645 T158 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=1,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=5}
   [junit4]   2> 99645 T139 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 99646 T158 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 99646 T158 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 99646 T158 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 99646 T139 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 99647 T139 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 99648 T158 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 99648 T158 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 99648 T139 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 99648 T158 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty4/index;done=false>>]
   [junit4]   2> 99649 T158 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty4/index
   [junit4]   2> 99649 T139 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 99649 T158 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty4;done=false>>]
   [junit4]   2> 99649 T139 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty3/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty3/index;done=false>>]
   [junit4]   2> 99650 T139 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty3/index
   [junit4]   2> 99650 T158 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty4
   [junit4]   2> 99650 T139 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty3 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty3;done=false>>]
   [junit4]   2> 99650 T158 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 99650 T139 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1393123554161/jetty3
   [junit4]   2> 99651 T158 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 99651 T139 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 99652 T139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 99652 T139 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 99652 T139 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=en_CA, timezone=Asia/Kuwait
   [junit4]   2> NOTE: Linux 3.8.0-35-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=24900184,total=64946176
   [junit4]   2> NOTE: All tests run in this JVM: [DocumentAnalysisResponseTest, SolrExceptionTest, ClientUtilsTest, LargeVolumeBinaryJettyTest, TestXMLEscaping, ContentStreamTest, CloudSolrServerTest]
   [junit4] Completed on J0 in 92.14s, 2 tests, 1 error <<< FAILURES!

[...truncated 121 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:453: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:433: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1274: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 48 suites, 280 tests, 1 error

Total time: 58 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9568 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9568/
Java: 32bit/jdk1.7.0_51 -server -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.client.solrj.impl.CloudSolrServerTest.testDistribSearch

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:47597 within 30000 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:47597 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([34590564DB11D59:82A31E4E3AEE7D65]:0)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:148)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:94)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:85)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:200)
	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:77)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:47597 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:222)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:140)
	... 47 more




Build Log:
[...truncated 11406 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
   [junit4]   2> 25154 T51 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 25155 T51 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 25157 T51 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
   [junit4]   2> 25163 T51 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1393144864858
   [junit4]   2> 25165 T51 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 25167 T52 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 25267 T51 oasc.ZkTestServer.run start zk server on port:47597
   [junit4]   2> 25302 T51 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 70157 T55 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 44832ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 70159 T51 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 70176 T53 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1445de7a0920000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 70178 T51 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:47597 47597
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrServerTest -Dtests.method=testDistribSearch -Dtests.seed=34590564DB11D59 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_US -Dtests.timezone=Asia/Jakarta -Dtests.file.encoding=UTF-8
   [junit4] ERROR   45.3s J0 | CloudSolrServerTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:47597 within 30000 ms
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([34590564DB11D59:82A31E4E3AEE7D65]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:148)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:94)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:85)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:200)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:77)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:47597 within 30000 ms
   [junit4]    > 	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:222)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:140)
   [junit4]    > 	... 47 more
   [junit4]   2> 70444 T51 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1393144910139
   [junit4]   2> 70444 T51 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 70445 T59 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 70545 T51 oasc.ZkTestServer.run start zk server on port:41876
   [junit4]   2> 70546 T51 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 70588 T65 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e70148 name:ZooKeeperConnection Watcher:127.0.0.1:41876 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 70589 T51 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 70591 T51 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 70613 T51 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 70616 T67 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b1c5c3 name:ZooKeeperConnection Watcher:127.0.0.1:41876/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 70616 T51 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 70622 T51 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 70627 T51 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 70631 T51 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 70634 T51 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 70638 T51 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 70648 T51 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 70656 T51 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 70657 T51 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 70661 T51 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 70662 T51 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
   [junit4]   2> 70662 T51 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
   [junit4]   2> 70663 T51 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
   [junit4]   2> 70663 T51 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 70664 T51 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 70664 T51 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 70665 T51 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 70665 T51 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 70675 T51 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 70677 T69 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@831c82 name:ZooKeeperConnection Watcher:127.0.0.1:41876/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 70677 T51 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 70687 T51 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 70689 T70 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01:0:0:0:0:0:0:114/ff01:0:0:0:0:0:0:114:33332
   [junit4]   2> 70689 T70 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.SocketException: Network is unreachable
   [junit4]   2> 	at sun.nio.ch.Net.connect0(Native Method)
   [junit4]   2> 	at sun.nio.ch.Net.connect(Net.java:465)
   [junit4]   2> 	at sun.nio.ch.Net.connect(Net.java:457)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:276)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:958)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 71791 T70 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01:0:0:0:0:0:0:114/ff01:0:0:0:0:0:0:114:33332
   [junit4]   2> 71893 T51 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
   [junit4]   2> 71895 T51 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:41876 41876
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 72106 T51 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 46958 T50 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=en_US, timezone=Asia/Jakarta
   [junit4]   2> NOTE: Linux 3.8.0-35-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=43214576,total=95420416
   [junit4]   2> NOTE: All tests run in this JVM: [TestJavaBinCodec, SolrExampleStreamingBinaryTest, CloudSolrServerTest]
   [junit4] Completed on J0 in 47.27s, 2 tests, 1 error <<< FAILURES!

[...truncated 136 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:453: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:433: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1274: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 48 suites, 280 tests, 1 error

Total time: 58 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -server -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0-fcs-b129) - Build # 9567 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9567/
Java: 32bit/jdk1.8.0-fcs-b129 -client -XX:+UseParallelGC

1 tests failed.
REGRESSION:  org.apache.solr.client.solrj.impl.CloudSolrServerTest.testDistribSearch

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:51751 within 30000 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:51751 within 30000 ms
	at __randomizedtesting.SeedInfo.seed([58C668EF604E5025:D920E6F717113019]:0)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:148)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:94)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:85)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89)
	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:200)
	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:77)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:771)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:51751 within 30000 ms
	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:222)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:140)
	... 47 more




Build Log:
[...truncated 11579 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
   [junit4]   2> 6008 T40 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 6009 T40 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 6011 T40 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
   [junit4]   2> 6019 T40 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1393137911803
   [junit4]   2> 6023 T40 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 6025 T41 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 6125 T40 oasc.ZkTestServer.run start zk server on port:51751
   [junit4]   2> 6164 T40 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 68652 T44 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 62471ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 68654 T40 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 68672 T42 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1445d7d883b0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 
   [junit4]   2> 68674 T40 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51751 51751
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudSolrServerTest -Dtests.method=testDistribSearch -Dtests.seed=58C668EF604E5025 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_PE -Dtests.timezone=Africa/Porto-Novo -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   62.8s J0 | CloudSolrServerTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:51751 within 30000 ms
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([58C668EF604E5025:D920E6F717113019]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:148)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:94)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:85)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89)
   [junit4]    > 	at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:200)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:77)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:744)
   [junit4]    > Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:51751 within 30000 ms
   [junit4]    > 	at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:222)
   [junit4]    > 	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:140)
   [junit4]    > 	... 47 more
   [junit4]   2> 68827 T40 oas.SolrTestCaseJ4.setUp ###Starting testShutdown
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1393137974610
   [junit4]   2> 68827 T40 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 68828 T48 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 68928 T40 oasc.ZkTestServer.run start zk server on port:42514
   [junit4]   2> 68929 T40 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 68934 T54 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d2d909 name:ZooKeeperConnection Watcher:127.0.0.1:42514 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 68935 T40 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 68936 T40 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 68957 T40 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 68959 T56 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e513c1 name:ZooKeeperConnection Watcher:127.0.0.1:42514/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 68959 T40 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 68964 T40 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 68968 T40 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 68972 T40 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 68975 T40 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 68979 T40 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 68986 T40 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 68992 T40 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 68993 T40 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 68998 T40 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 68999 T40 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
   [junit4]   2> 68999 T40 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
   [junit4]   2> 69000 T40 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
   [junit4]   2> 69001 T40 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 69001 T40 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 69002 T40 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 69002 T40 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 69003 T40 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 69009 T40 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 69011 T58 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b068fb name:ZooKeeperConnection Watcher:127.0.0.1:42514/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 69012 T40 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 69020 T40 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 69022 T59 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01:0:0:0:0:0:0:114/ff01:0:0:0:0:0:0:114:33332
   [junit4]   2> 69022 T59 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.SocketException: Network is unreachable
   [junit4]   2> 	at sun.nio.ch.Net.connect0(Native Method)
   [junit4]   2> 	at sun.nio.ch.Net.connect(Net.java:435)
   [junit4]   2> 	at sun.nio.ch.Net.connect(Net.java:427)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:643)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:276)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:958)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
   [junit4]   2> 
   [junit4]   2> 70124 T59 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01:0:0:0:0:0:0:114/ff01:0:0:0:0:0:0:114:33332
   [junit4]   2> 70227 T40 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
   [junit4]   2> 70228 T40 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:42514 42514
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 70431 T40 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 64429 T39 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=es_PE, timezone=Africa/Porto-Novo
   [junit4]   2> NOTE: Linux 3.8.0-35-generic i386/Oracle Corporation 1.8.0 (32-bit)/cpus=8,threads=1,free=21388248,total=195559424
   [junit4]   2> NOTE: All tests run in this JVM: [LargeVolumeJettyTest, TestJavaBinCodec, CloudSolrServerTest]
   [junit4] Completed on J0 in 65.52s, 2 tests, 1 error <<< FAILURES!

[...truncated 136 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:453: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:433: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1274: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:907: There were test failures: 48 suites, 280 tests, 1 error

Total time: 50 minutes 31 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-fcs-b129 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_51) - Build # 9566 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/9566/
Java: 32bit/jdk1.7.0_51 -server -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 50353 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:453: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:392: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:87: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:185: The following files are missing svn:eol-style (or binary svn:mime-type):
* ./solr/licenses/jcl-over-slf4j-1.7.6.jar.sha1
* ./solr/licenses/jul-to-slf4j-1.7.6.jar.sha1
* ./solr/licenses/slf4j-api-1.7.6.jar.sha1
* ./solr/licenses/slf4j-log4j12-1.7.6.jar.sha1

Total time: 66 minutes 10 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -server -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure