You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/17 20:01:29 UTC
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_21) - Build #
6179 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6179/
Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseParallelGC
1 tests failed.
REGRESSION: org.apache.solr.client.solrj.impl.CloudSolrServerTest.testDistribSearch
Error Message:
Stack Trace:
java.lang.NullPointerException
at __randomizedtesting.SeedInfo.seed([739BCB3D49E058B1:F27D45253EBF388D]:0)
at org.apache.solr.client.solrj.impl.CloudSolrServerTest.doTest(CloudSolrServerTest.java:108)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 10476 lines...]
[junit4:junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
[junit4:junit4] 1> 15155 [SUITE-CloudSolrServerTest-seed#[739BCB3D49E058B1]-worker] INFO org.apache.solr.BaseDistributedSearchTestCase ? Setting hostContext system property: /
[junit4:junit4] 1> 15157 [SUITE-CloudSolrServerTest-seed#[739BCB3D49E058B1]-worker] WARN org.apache.solr.cloud.AbstractZkTestCase ? TEST_HOME() does not exist - solrj test?
[junit4:junit4] 1> 15162 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.SolrTestCaseJ4 ? ###Starting testDistribSearch
[junit4:junit4] 1> 15164 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkTestServer ? STARTING ZK TEST SERVER
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> 15167 [Thread-59] INFO org.apache.solr.cloud.ZkTestServer ? Starting server
[junit4:junit4] 1> 15266 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkTestServer ? start zk server on port:58974
[junit4:junit4] 1> 15300 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> 15341 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager ? Watcher org.apache.solr.common.cloud.ConnectionManager@21805932 name:ZooKeeperConnection Watcher:127.0.0.1:58974 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> 15342 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Client is connected to ZooKeeper
[junit4:junit4] 1> 15343 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /solr
[junit4:junit4] 1> 15382 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> 15386 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager ? Watcher org.apache.solr.common.cloud.ConnectionManager@26bf49c name:ZooKeeperConnection Watcher:127.0.0.1:58974/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> 15387 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Client is connected to ZooKeeper
[junit4:junit4] 1> 15395 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /collections/collection1
[junit4:junit4] 1> 15414 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /collections/collection1/shards
[junit4:junit4] 1> 15424 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /collections/control_collection
[junit4:junit4] 1> 15432 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /collections/control_collection/shards
[junit4:junit4] 1> 15441 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.AbstractZkTestCase ? put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 1> 15454 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 1> 15467 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.AbstractZkTestCase ? put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 1> 15468 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /configs/conf1/schema.xml
[junit4:junit4] 1> 15482 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.AbstractZkTestCase ? skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
[junit4:junit4] 1> 15482 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.AbstractZkTestCase ? skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
[junit4:junit4] 1> 15483 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.AbstractZkTestCase ? skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
[junit4:junit4] 1> 15483 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.AbstractZkTestCase ? skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
[junit4:junit4] 1> 15484 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.AbstractZkTestCase ? skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
[junit4:junit4] 1> 15484 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.AbstractZkTestCase ? skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
[junit4:junit4] 1> 15485 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.AbstractZkTestCase ? skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
[junit4:junit4] 1> 15518 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.eclipse.jetty.server.Server ? jetty-8.1.10.v20130312
[junit4:junit4] 1> 15521 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.eclipse.jetty.server.AbstractConnector ? Started SelectChannelConnector@127.0.0.1:47252
[junit4:junit4] 1> 15521 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.servlet.SolrDispatchFilter ? SolrDispatchFilter.init()
[junit4:junit4] 1> 15522 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrResourceLoader ? JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 1> 15522 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrResourceLoader ? using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1371492012259
[junit4:junit4] 1> 15523 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1371492012259/solr.xml
[junit4:junit4] 1> 15523 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? New CoreContainer 276280784
[junit4:junit4] 1> 15524 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1371492012259/'
[junit4:junit4] 1> 15524 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrResourceLoader ? new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1371492012259/'
[junit4:junit4] 1> 15591 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting socketTimeout to: 0
[junit4:junit4] 1> 15592 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting urlScheme to: http://
[junit4:junit4] 1> 15593 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting connTimeout to: 0
[junit4:junit4] 1> 15593 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting maxConnectionsPerHost to: 20
[junit4:junit4] 1> 15593 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting corePoolSize to: 0
[junit4:junit4] 1> 15594 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting maximumPoolSize to: 2147483647
[junit4:junit4] 1> 15594 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting maxThreadIdleTime to: 5
[junit4:junit4] 1> 15595 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting sizeOfQueue to: -1
[junit4:junit4] 1> 15595 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting fairnessPolicy to: false
[junit4:junit4] 1> 15596 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4] 1> 15640 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.logging.LogWatcher ? Registering Log Listener
[junit4:junit4] 1> 15640 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.ZkContainer ? Zookeeper client=127.0.0.1:58974/solr
[junit4:junit4] 1> 15648 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkController ? zkHost includes chroot
[junit4:junit4] 1> 15649 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> 15652 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager ? Watcher org.apache.solr.common.cloud.ConnectionManager@5bbfbb2f name:ZooKeeperConnection Watcher:127.0.0.1:58974 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> 15653 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Client is connected to ZooKeeper
[junit4:junit4] 1> 15656 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=0&connTimeout=0
[junit4:junit4] 1> 15661 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> 15663 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager ? Watcher org.apache.solr.common.cloud.ConnectionManager@26c70ebb name:ZooKeeperConnection Watcher:127.0.0.1:58974/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> 15663 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Client is connected to ZooKeeper
[junit4:junit4] 1> 15668 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /overseer/queue
[junit4:junit4] 1> 15679 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /overseer/collection-queue-work
[junit4:junit4] 1> 15691 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /live_nodes
[junit4:junit4] 1> 15695 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkController ? Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47252_
[junit4:junit4] 1> 15697 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /live_nodes/127.0.0.1:47252_
[junit4:junit4] 1> 15703 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /overseer_elect/election
[junit4:junit4] 1> 15714 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /overseer_elect/leader
[junit4:junit4] 1> 15718 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.Overseer ? Overseer (id=89882100497645571-127.0.0.1:47252_-n_0000000000) starting
[junit4:junit4] 1> 15733 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /overseer/queue-work
[junit4:junit4] 1> 15751 [Overseer-89882100497645571-127.0.0.1:47252_-n_0000000000] INFO org.apache.solr.cloud.OverseerCollectionProcessor ? Process current queue of collection creations
[junit4:junit4] 1> 15752 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /clusterstate.json
[junit4:junit4] 1> 15764 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /aliases.json
[junit4:junit4] 1> 15771 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cluster state from ZooKeeper...
[junit4:junit4] 1> 15781 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Starting to work on the main queue
[junit4:junit4] 1> 15786 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.ZkController ? publishing core=collection1 state=down
[junit4:junit4] 1> 15787 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.ZkController ? numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> 17292 [Thread-69] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 17295 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Update state numShards=1 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":null,
[junit4:junit4] 1> "numShards":"1",
[junit4:junit4] 1> "shard":null,
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:47252_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:47252"}
[junit4:junit4] 1> 17296 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Create collection control_collection with numShards 1
[junit4:junit4] 1> 17308 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Assigning new node to shard=shard1
[junit4:junit4] 1> 17325 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> 17789 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.CoreContainer ? Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1371492012259/collection1
[junit4:junit4] 1> 17790 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.ZkController ? Check for collection zkNode:control_collection
[junit4:junit4] 1> 17791 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.ZkController ? Collection zkNode exists
[junit4:junit4] 1> 17791 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.ZkController ? Load collection config from:/collections/control_collection
[junit4:junit4] 1> 17794 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.SolrResourceLoader ? new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1371492012259/collection1/'
[junit4:junit4] 1> 17827 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.SolrConfig ? Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 1> 17869 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.Config ? Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 1> 17871 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.schema.IndexSchema ? Reading Solr Schema from schema.xml
[junit4:junit4] 1> 17902 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.schema.IndexSchema ? [collection1] Schema name=test
[junit4:junit4] 1> 18339 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.schema.IndexSchema ? default search field in schema is text
[junit4:junit4] 1> 18342 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.schema.IndexSchema ? unique key field: id
[junit4:junit4] 1> 18344 [coreLoadExecutor-26-thread-1] WARN org.apache.solr.schema.IndexSchema ? Field text is not multivalued and destination for multiple copyFields (3)
[junit4:junit4] 1> 18345 [coreLoadExecutor-26-thread-1] WARN org.apache.solr.schema.IndexSchema ? Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
[junit4:junit4] 1> 18348 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.SolrCore ? org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> 18348 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.SolrCore ? [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-controljetty-1371492012259/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/control/data/
[junit4:junit4] 1> 18349 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.SolrCore ? JMX monitoring not detected for core: collection1
[junit4:junit4] 1> 18349 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.SolrCore ? org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> 18350 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.CachingDirectoryFactory ? return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/control/data
[junit4:junit4] 1> 18350 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.SolrCore ? New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/control/data/index/
[junit4:junit4] 1> 18350 [coreLoadExecutor-26-thread-1] WARN org.apache.solr.core.SolrCore ? [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 1> 18351 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.CachingDirectoryFactory ? return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/control/data/index
[junit4:junit4] 1> 18352 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7bfea51e lockFactory=org.apache.lucene.store.NativeFSLockFactory@57ed07c5),segFN=segments_1,generation=1}
[junit4:junit4] 1> 18353 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.SolrCore ? newest commit generation = 1
[junit4:junit4] 1> 18354 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /get: solr.RealTimeGetHandler
[junit4:junit4] 1> 18354 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.RequestHandlers ? adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 1> 18354 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /replication: solr.ReplicationHandler
[junit4:junit4] 1> 18359 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.RequestHandlers ? created standard: solr.StandardRequestHandler
[junit4:junit4] 1> 18359 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /update: solr.UpdateRequestHandler
[junit4:junit4] 1> 18359 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 1> 18360 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /admin/ping: solr.PingRequestHandler
[junit4:junit4] 1> 18362 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.handler.loader.XMLLoader ? xsltCacheLifetimeSeconds=60
[junit4:junit4] 1> 18365 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.search.SolrIndexSearcher ? Opening Searcher@3b9771a5 main
[junit4:junit4] 1> 18365 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.update.UpdateHandler ? Clearing tlog files, tlogDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/control/data/tlog
[junit4:junit4] 1> 18366 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.SolrCore ? Hard AutoCommit: disabled
[junit4:junit4] 1> 18366 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.SolrCore ? Soft AutoCommit: disabled
[junit4:junit4] 1> 18368 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.core.CoreContainer ? registering core: collection1
[junit4:junit4] 1> 18368 [searcherExecutor-27-thread-1] INFO org.apache.solr.core.SolrCore ? [collection1] Registered new searcher Searcher@3b9771a5 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 1> 18368 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.ZkController ? Register replica - core:collection1 address:http://127.0.0.1:47252 collection:control_collection shard:shard1
[junit4:junit4] 1> 18373 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 1> 18382 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 1> 18396 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext ? Running the leader process.
[junit4:junit4] 1> 18403 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext ? Enough replicas found to continue.
[junit4:junit4] 1> 18403 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext ? I may be the new leader - try and sync
[junit4:junit4] 1> 18404 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.SyncStrategy ? Sync replicas to http://127.0.0.1:47252/collection1/
[junit4:junit4] 1> 18404 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.SyncStrategy ? Sync Success - now sync replicas to me
[junit4:junit4] 1> 18405 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.SyncStrategy ? http://127.0.0.1:47252/collection1/ has no replicas
[junit4:junit4] 1> 18405 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext ? I am the new leader: http://127.0.0.1:47252/collection1/
[junit4:junit4] 1> 18405 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 1> 18834 [Thread-69] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 18871 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> 18918 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.ZkController ? We are http://127.0.0.1:47252/collection1/ and leader is http://127.0.0.1:47252/collection1/
[junit4:junit4] 1> 18919 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.ZkController ? No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47252
[junit4:junit4] 1> 18920 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.ZkController ? I am the leader, no recovery necessary
[junit4:junit4] 1> 18920 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.ZkController ? publishing core=collection1 state=active
[junit4:junit4] 1> 18920 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.cloud.ZkController ? numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> 18923 [coreLoadExecutor-26-thread-1] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 18925 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.servlet.SolrDispatchFilter ? user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
[junit4:junit4] 1> 18926 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.servlet.SolrDispatchFilter ? SolrDispatchFilter.init() done
[junit4:junit4] 1> 18926 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> 18930 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:
[junit4:junit4] 1> 18933 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> 18935 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager ? Watcher org.apache.solr.common.cloud.ConnectionManager@19cc914e name:ZooKeeperConnection Watcher:127.0.0.1:58974/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> 18936 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Client is connected to ZooKeeper
[junit4:junit4] 1> 18938 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cluster state from ZooKeeper...
[junit4:junit4] 1> 18943 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ChaosMonkey ? monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 1> 18982 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.eclipse.jetty.server.Server ? jetty-8.1.10.v20130312
[junit4:junit4] 1> 18985 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.eclipse.jetty.server.AbstractConnector ? Started SelectChannelConnector@127.0.0.1:38962
[junit4:junit4] 1> 18986 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.servlet.SolrDispatchFilter ? SolrDispatchFilter.init()
[junit4:junit4] 1> 18987 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrResourceLoader ? JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 1> 18987 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrResourceLoader ? using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1371492015715
[junit4:junit4] 1> 18988 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1371492015715/solr.xml
[junit4:junit4] 1> 18989 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? New CoreContainer 777887041
[junit4:junit4] 1> 18989 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1371492015715/'
[junit4:junit4] 1> 18990 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrResourceLoader ? new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1371492015715/'
[junit4:junit4] 1> 19057 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting socketTimeout to: 0
[junit4:junit4] 1> 19057 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting urlScheme to: http://
[junit4:junit4] 1> 19058 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting connTimeout to: 0
[junit4:junit4] 1> 19058 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting maxConnectionsPerHost to: 20
[junit4:junit4] 1> 19059 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting corePoolSize to: 0
[junit4:junit4] 1> 19059 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting maximumPoolSize to: 2147483647
[junit4:junit4] 1> 19059 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting maxThreadIdleTime to: 5
[junit4:junit4] 1> 19060 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting sizeOfQueue to: -1
[junit4:junit4] 1> 19060 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting fairnessPolicy to: false
[junit4:junit4] 1> 19061 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4] 1> 19065 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.logging.LogWatcher ? Registering Log Listener
[junit4:junit4] 1> 19065 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.ZkContainer ? Zookeeper client=127.0.0.1:58974/solr
[junit4:junit4] 1> 19066 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkController ? zkHost includes chroot
[junit4:junit4] 1> 19067 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> 19069 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager ? Watcher org.apache.solr.common.cloud.ConnectionManager@636e6f5e name:ZooKeeperConnection Watcher:127.0.0.1:58974 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> 19069 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Client is connected to ZooKeeper
[junit4:junit4] 1> 19072 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=0&connTimeout=0
[junit4:junit4] 1> 19079 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> 19081 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager ? Watcher org.apache.solr.common.cloud.ConnectionManager@4a7f4d2c name:ZooKeeperConnection Watcher:127.0.0.1:58974/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> 19081 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Client is connected to ZooKeeper
[junit4:junit4] 1> 19087 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cluster state from ZooKeeper...
[junit4:junit4] 1> 20092 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkController ? Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38962_
[junit4:junit4] 1> 20094 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /live_nodes/127.0.0.1:38962_
[junit4:junit4] 1> 20098 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> 20099 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (2)
[junit4:junit4] 1> 20100 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (2)
[junit4:junit4] 1> 20101 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (2)
[junit4:junit4] 1> 20108 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.ZkController ? publishing core=collection1 state=down
[junit4:junit4] 1> 20108 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.ZkController ? numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> 20390 [Thread-69] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 20392 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Update state numShards=1 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"127.0.0.1:47252__collection1",
[junit4:junit4] 1> "numShards":"1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:47252_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:47252"}
[junit4:junit4] 1> 20398 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":null,
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":null,
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:38962_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:38962"}
[junit4:junit4] 1> 20398 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Create collection collection1 with numShards 2
[junit4:junit4] 1> 20398 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Assigning new node to shard=shard1
[junit4:junit4] 1> 20491 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> 20491 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> 20491 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> 21110 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.CoreContainer ? Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1371492015715/collection1
[junit4:junit4] 1> 21111 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.ZkController ? Check for collection zkNode:collection1
[junit4:junit4] 1> 21112 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.ZkController ? Collection zkNode exists
[junit4:junit4] 1> 21113 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.ZkController ? Load collection config from:/collections/collection1
[junit4:junit4] 1> 21117 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.SolrResourceLoader ? new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1371492015715/collection1/'
[junit4:junit4] 1> 21157 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.SolrConfig ? Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 1> 21208 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.Config ? Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 1> 21211 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.schema.IndexSchema ? Reading Solr Schema from schema.xml
[junit4:junit4] 1> 21227 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.schema.IndexSchema ? [collection1] Schema name=test
[junit4:junit4] 1> 21569 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.schema.IndexSchema ? default search field in schema is text
[junit4:junit4] 1> 21571 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.schema.IndexSchema ? unique key field: id
[junit4:junit4] 1> 21573 [coreLoadExecutor-36-thread-1] WARN org.apache.solr.schema.IndexSchema ? Field text is not multivalued and destination for multiple copyFields (3)
[junit4:junit4] 1> 21574 [coreLoadExecutor-36-thread-1] WARN org.apache.solr.schema.IndexSchema ? Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
[junit4:junit4] 1> 21575 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.SolrCore ? org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> 21575 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.SolrCore ? [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty1-1371492015715/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty1/
[junit4:junit4] 1> 21576 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.SolrCore ? JMX monitoring not detected for core: collection1
[junit4:junit4] 1> 21576 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.SolrCore ? org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> 21576 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.CachingDirectoryFactory ? return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty1
[junit4:junit4] 1> 21577 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.SolrCore ? New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty1/index/
[junit4:junit4] 1> 21577 [coreLoadExecutor-36-thread-1] WARN org.apache.solr.core.SolrCore ? [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 1> 21577 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.CachingDirectoryFactory ? return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty1/index
[junit4:junit4] 1> 21579 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@b836456 lockFactory=org.apache.lucene.store.NativeFSLockFactory@bf6e58c),segFN=segments_1,generation=1}
[junit4:junit4] 1> 21579 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.SolrCore ? newest commit generation = 1
[junit4:junit4] 1> 21580 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /get: solr.RealTimeGetHandler
[junit4:junit4] 1> 21581 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.RequestHandlers ? adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 1> 21581 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /replication: solr.ReplicationHandler
[junit4:junit4] 1> 21581 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.RequestHandlers ? created standard: solr.StandardRequestHandler
[junit4:junit4] 1> 21582 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /update: solr.UpdateRequestHandler
[junit4:junit4] 1> 21582 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 1> 21583 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /admin/ping: solr.PingRequestHandler
[junit4:junit4] 1> 21585 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.handler.loader.XMLLoader ? xsltCacheLifetimeSeconds=60
[junit4:junit4] 1> 21588 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.search.SolrIndexSearcher ? Opening Searcher@71955e74 main
[junit4:junit4] 1> 21588 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.update.UpdateHandler ? Clearing tlog files, tlogDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty1/tlog
[junit4:junit4] 1> 21589 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.SolrCore ? Hard AutoCommit: disabled
[junit4:junit4] 1> 21589 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.SolrCore ? Soft AutoCommit: disabled
[junit4:junit4] 1> 21597 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.core.CoreContainer ? registering core: collection1
[junit4:junit4] 1> 21597 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.ZkController ? Register replica - core:collection1 address:http://127.0.0.1:38962 collection:collection1 shard:shard1
[junit4:junit4] 1> 21597 [searcherExecutor-37-thread-1] INFO org.apache.solr.core.SolrCore ? [collection1] Registered new searcher Searcher@71955e74 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 1> 21599 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 1> 21610 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext ? Running the leader process.
[junit4:junit4] 1> 21614 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext ? Enough replicas found to continue.
[junit4:junit4] 1> 21615 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext ? I may be the new leader - try and sync
[junit4:junit4] 1> 21616 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.SyncStrategy ? Sync replicas to http://127.0.0.1:38962/collection1/
[junit4:junit4] 1> 21617 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.SyncStrategy ? Sync Success - now sync replicas to me
[junit4:junit4] 1> 21618 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.SyncStrategy ? http://127.0.0.1:38962/collection1/ has no replicas
[junit4:junit4] 1> 21618 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext ? I am the new leader: http://127.0.0.1:38962/collection1/
[junit4:junit4] 1> 21619 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 1> 22006 [Thread-69] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 22027 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> 22027 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> 22027 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> 22036 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.ZkController ? We are http://127.0.0.1:38962/collection1/ and leader is http://127.0.0.1:38962/collection1/
[junit4:junit4] 1> 22037 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.ZkController ? No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38962
[junit4:junit4] 1> 22037 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.ZkController ? I am the leader, no recovery necessary
[junit4:junit4] 1> 22037 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.ZkController ? publishing core=collection1 state=active
[junit4:junit4] 1> 22037 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.cloud.ZkController ? numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> 22040 [coreLoadExecutor-36-thread-1] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 22042 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.servlet.SolrDispatchFilter ? user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
[junit4:junit4] 1> 22042 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.servlet.SolrDispatchFilter ? SolrDispatchFilter.init() done
[junit4:junit4] 1> 22043 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> 22073 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.eclipse.jetty.server.Server ? jetty-8.1.10.v20130312
[junit4:junit4] 1> 22075 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.eclipse.jetty.server.AbstractConnector ? Started SelectChannelConnector@127.0.0.1:59478
[junit4:junit4] 1> 22077 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.servlet.SolrDispatchFilter ? SolrDispatchFilter.init()
[junit4:junit4] 1> 22077 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrResourceLoader ? JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 1> 22078 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrResourceLoader ? using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1371492018820
[junit4:junit4] 1> 22078 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1371492018820/solr.xml
[junit4:junit4] 1> 22078 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? New CoreContainer 742976600
[junit4:junit4] 1> 22079 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1371492018820/'
[junit4:junit4] 1> 22079 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrResourceLoader ? new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1371492018820/'
[junit4:junit4] 1> 22130 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting socketTimeout to: 0
[junit4:junit4] 1> 22131 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting urlScheme to: http://
[junit4:junit4] 1> 22131 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting connTimeout to: 0
[junit4:junit4] 1> 22131 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting maxConnectionsPerHost to: 20
[junit4:junit4] 1> 22132 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting corePoolSize to: 0
[junit4:junit4] 1> 22132 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting maximumPoolSize to: 2147483647
[junit4:junit4] 1> 22133 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting maxThreadIdleTime to: 5
[junit4:junit4] 1> 22133 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting sizeOfQueue to: -1
[junit4:junit4] 1> 22133 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting fairnessPolicy to: false
[junit4:junit4] 1> 22134 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4] 1> 22138 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.logging.LogWatcher ? Registering Log Listener
[junit4:junit4] 1> 22138 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.ZkContainer ? Zookeeper client=127.0.0.1:58974/solr
[junit4:junit4] 1> 22139 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkController ? zkHost includes chroot
[junit4:junit4] 1> 22139 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> 22144 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager ? Watcher org.apache.solr.common.cloud.ConnectionManager@2f34af37 name:ZooKeeperConnection Watcher:127.0.0.1:58974 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> 22144 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Client is connected to ZooKeeper
[junit4:junit4] 1> 22147 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=0&connTimeout=0
[junit4:junit4] 1> 22151 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> 22153 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager ? Watcher org.apache.solr.common.cloud.ConnectionManager@364dabbb name:ZooKeeperConnection Watcher:127.0.0.1:58974/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> 22153 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Client is connected to ZooKeeper
[junit4:junit4] 1> 22160 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cluster state from ZooKeeper...
[junit4:junit4] 1> 23165 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkController ? Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59478_
[junit4:junit4] 1> 23168 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /live_nodes/127.0.0.1:59478_
[junit4:junit4] 1> 23174 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> 23175 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> 23175 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> 23176 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (3)
[junit4:junit4] 1> 23180 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (3)
[junit4:junit4] 1> 23182 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (3)
[junit4:junit4] 1> 23182 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (3)
[junit4:junit4] 1> 23190 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.ZkController ? publishing core=collection1 state=down
[junit4:junit4] 1> 23191 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.ZkController ? numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> 23539 [Thread-69] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 23542 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"127.0.0.1:38962__collection1",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:38962_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:38962"}
[junit4:junit4] 1> 23552 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":null,
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":null,
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:59478_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:59478"}
[junit4:junit4] 1> 23553 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Collection already exists with numShards=2
[junit4:junit4] 1> 23554 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Assigning new node to shard=shard2
[junit4:junit4] 1> 23564 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> 23565 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> 23565 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> 23564 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> 24193 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.CoreContainer ? Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1371492018820/collection1
[junit4:junit4] 1> 24194 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.ZkController ? Check for collection zkNode:collection1
[junit4:junit4] 1> 24196 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.ZkController ? Collection zkNode exists
[junit4:junit4] 1> 24196 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.ZkController ? Load collection config from:/collections/collection1
[junit4:junit4] 1> 24200 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.SolrResourceLoader ? new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1371492018820/collection1/'
[junit4:junit4] 1> 24242 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.SolrConfig ? Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 1> 24279 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.Config ? Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 1> 24281 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.schema.IndexSchema ? Reading Solr Schema from schema.xml
[junit4:junit4] 1> 24297 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.schema.IndexSchema ? [collection1] Schema name=test
[junit4:junit4] 1> 24619 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.schema.IndexSchema ? default search field in schema is text
[junit4:junit4] 1> 24622 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.schema.IndexSchema ? unique key field: id
[junit4:junit4] 1> 24624 [coreLoadExecutor-46-thread-1] WARN org.apache.solr.schema.IndexSchema ? Field text is not multivalued and destination for multiple copyFields (3)
[junit4:junit4] 1> 24624 [coreLoadExecutor-46-thread-1] WARN org.apache.solr.schema.IndexSchema ? Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
[junit4:junit4] 1> 24626 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.SolrCore ? org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> 24626 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.SolrCore ? [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty2-1371492018820/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty2/
[junit4:junit4] 1> 24626 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.SolrCore ? JMX monitoring not detected for core: collection1
[junit4:junit4] 1> 24626 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.SolrCore ? org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> 24627 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.CachingDirectoryFactory ? return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty2
[junit4:junit4] 1> 24627 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.SolrCore ? New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty2/index/
[junit4:junit4] 1> 24628 [coreLoadExecutor-46-thread-1] WARN org.apache.solr.core.SolrCore ? [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 1> 24628 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.CachingDirectoryFactory ? return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty2/index
[junit4:junit4] 1> 24630 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7e1b4e9d lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f7b8f60),segFN=segments_1,generation=1}
[junit4:junit4] 1> 24630 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.SolrCore ? newest commit generation = 1
[junit4:junit4] 1> 24631 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /get: solr.RealTimeGetHandler
[junit4:junit4] 1> 24632 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.RequestHandlers ? adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 1> 24632 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /replication: solr.ReplicationHandler
[junit4:junit4] 1> 24632 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.RequestHandlers ? created standard: solr.StandardRequestHandler
[junit4:junit4] 1> 24633 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /update: solr.UpdateRequestHandler
[junit4:junit4] 1> 24633 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 1> 24633 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /admin/ping: solr.PingRequestHandler
[junit4:junit4] 1> 24636 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.handler.loader.XMLLoader ? xsltCacheLifetimeSeconds=60
[junit4:junit4] 1> 24639 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.search.SolrIndexSearcher ? Opening Searcher@297978c5 main
[junit4:junit4] 1> 24639 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.update.UpdateHandler ? Clearing tlog files, tlogDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty2/tlog
[junit4:junit4] 1> 24640 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.SolrCore ? Hard AutoCommit: disabled
[junit4:junit4] 1> 24640 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.SolrCore ? Soft AutoCommit: disabled
[junit4:junit4] 1> 24641 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.core.CoreContainer ? registering core: collection1
[junit4:junit4] 1> 24642 [searcherExecutor-47-thread-1] INFO org.apache.solr.core.SolrCore ? [collection1] Registered new searcher Searcher@297978c5 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 1> 24642 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.ZkController ? Register replica - core:collection1 address:http://127.0.0.1:59478 collection:collection1 shard:shard2
[junit4:junit4] 1> 24644 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 1> 24655 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext ? Running the leader process.
[junit4:junit4] 1> 24661 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext ? Enough replicas found to continue.
[junit4:junit4] 1> 24661 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext ? I may be the new leader - try and sync
[junit4:junit4] 1> 24662 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.SyncStrategy ? Sync replicas to http://127.0.0.1:59478/collection1/
[junit4:junit4] 1> 24662 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.SyncStrategy ? Sync Success - now sync replicas to me
[junit4:junit4] 1> 24662 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.SyncStrategy ? http://127.0.0.1:59478/collection1/ has no replicas
[junit4:junit4] 1> 24663 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.ShardLeaderElectionContext ? I am the new leader: http://127.0.0.1:59478/collection1/
[junit4:junit4] 1> 24663 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 1> 25076 [Thread-69] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 25088 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> 25088 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> 25088 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> 25088 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> 25124 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.ZkController ? We are http://127.0.0.1:59478/collection1/ and leader is http://127.0.0.1:59478/collection1/
[junit4:junit4] 1> 25125 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.ZkController ? No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59478
[junit4:junit4] 1> 25125 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.ZkController ? I am the leader, no recovery necessary
[junit4:junit4] 1> 25125 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.ZkController ? publishing core=collection1 state=active
[junit4:junit4] 1> 25126 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.cloud.ZkController ? numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> 25128 [coreLoadExecutor-46-thread-1] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 25129 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.servlet.SolrDispatchFilter ? user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
[junit4:junit4] 1> 25130 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.servlet.SolrDispatchFilter ? SolrDispatchFilter.init() done
[junit4:junit4] 1> 25130 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> 25162 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.eclipse.jetty.server.Server ? jetty-8.1.10.v20130312
[junit4:junit4] 1> 25165 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.eclipse.jetty.server.AbstractConnector ? Started SelectChannelConnector@127.0.0.1:37922
[junit4:junit4] 1> 25166 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.servlet.SolrDispatchFilter ? SolrDispatchFilter.init()
[junit4:junit4] 1> 25166 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrResourceLoader ? JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 1> 25166 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrResourceLoader ? using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1371492021907
[junit4:junit4] 1> 25167 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1371492021907/solr.xml
[junit4:junit4] 1> 25167 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? New CoreContainer 308779054
[junit4:junit4] 1> 25168 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1371492021907/'
[junit4:junit4] 1> 25168 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrResourceLoader ? new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1371492021907/'
[junit4:junit4] 1> 25220 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting socketTimeout to: 0
[junit4:junit4] 1> 25221 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting urlScheme to: http://
[junit4:junit4] 1> 25221 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting connTimeout to: 0
[junit4:junit4] 1> 25222 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting maxConnectionsPerHost to: 20
[junit4:junit4] 1> 25222 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting corePoolSize to: 0
[junit4:junit4] 1> 25223 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting maximumPoolSize to: 2147483647
[junit4:junit4] 1> 25223 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting maxThreadIdleTime to: 5
[junit4:junit4] 1> 25223 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting sizeOfQueue to: -1
[junit4:junit4] 1> 25224 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting fairnessPolicy to: false
[junit4:junit4] 1> 25224 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4] 1> 25228 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.logging.LogWatcher ? Registering Log Listener
[junit4:junit4] 1> 25229 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.ZkContainer ? Zookeeper client=127.0.0.1:58974/solr
[junit4:junit4] 1> 25231 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkController ? zkHost includes chroot
[junit4:junit4] 1> 25233 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> 25235 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager ? Watcher org.apache.solr.common.cloud.ConnectionManager@5316c9ca name:ZooKeeperConnection Watcher:127.0.0.1:58974 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> 25236 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Client is connected to ZooKeeper
[junit4:junit4] 1> 25238 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=0&connTimeout=0
[junit4:junit4] 1> 25244 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> 25246 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager ? Watcher org.apache.solr.common.cloud.ConnectionManager@11718107 name:ZooKeeperConnection Watcher:127.0.0.1:58974/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> 25246 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Client is connected to ZooKeeper
[junit4:junit4] 1> 25253 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cluster state from ZooKeeper...
[junit4:junit4] 1> 26259 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkController ? Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37922_
[junit4:junit4] 1> 26262 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /live_nodes/127.0.0.1:37922_
[junit4:junit4] 1> 26268 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> 26268 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> 26268 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> 26270 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (4)
[junit4:junit4] 1> 26271 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 26270 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (4)
[junit4:junit4] 1> 26274 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (4)
[junit4:junit4] 1> 26275 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (4)
[junit4:junit4] 1> 26276 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (4)
[junit4:junit4] 1> 26287 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.cloud.ZkController ? publishing core=collection1 state=down
[junit4:junit4] 1> 26288 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.cloud.ZkController ? numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> 26597 [Thread-69] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 26599 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"127.0.0.1:59478__collection1",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:59478_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:59478"}
[junit4:junit4] 1> 26605 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":null,
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":null,
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:37922_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:37922"}
[junit4:junit4] 1> 26605 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Collection already exists with numShards=2
[junit4:junit4] 1> 26605 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Assigning new node to shard=shard1
[junit4:junit4] 1> 26611 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 26612 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 26612 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 26612 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 26611 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 27294 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.CoreContainer ? Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1371492021907/collection1
[junit4:junit4] 1> 27295 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.cloud.ZkController ? Check for collection zkNode:collection1
[junit4:junit4] 1> 27297 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.cloud.ZkController ? Collection zkNode exists
[junit4:junit4] 1> 27298 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.cloud.ZkController ? Load collection config from:/collections/collection1
[junit4:junit4] 1> 27302 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.SolrResourceLoader ? new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1371492021907/collection1/'
[junit4:junit4] 1> 27344 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.SolrConfig ? Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 1> 27390 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.Config ? Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 1> 27392 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.schema.IndexSchema ? Reading Solr Schema from schema.xml
[junit4:junit4] 1> 27409 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.schema.IndexSchema ? [collection1] Schema name=test
[junit4:junit4] 1> 27780 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.schema.IndexSchema ? default search field in schema is text
[junit4:junit4] 1> 27782 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.schema.IndexSchema ? unique key field: id
[junit4:junit4] 1> 27784 [coreLoadExecutor-56-thread-1] WARN org.apache.solr.schema.IndexSchema ? Field text is not multivalued and destination for multiple copyFields (3)
[junit4:junit4] 1> 27785 [coreLoadExecutor-56-thread-1] WARN org.apache.solr.schema.IndexSchema ? Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
[junit4:junit4] 1> 27786 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.SolrCore ? org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> 27786 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.SolrCore ? [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty3-1371492021907/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty3/
[junit4:junit4] 1> 27787 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.SolrCore ? JMX monitoring not detected for core: collection1
[junit4:junit4] 1> 27787 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.SolrCore ? org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> 27788 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.CachingDirectoryFactory ? return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty3
[junit4:junit4] 1> 27789 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.SolrCore ? New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty3/index/
[junit4:junit4] 1> 27789 [coreLoadExecutor-56-thread-1] WARN org.apache.solr.core.SolrCore ? [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 1> 27790 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.CachingDirectoryFactory ? return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty3/index
[junit4:junit4] 1> 27792 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4c8796c7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a5104f6),segFN=segments_1,generation=1}
[junit4:junit4] 1> 27792 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.SolrCore ? newest commit generation = 1
[junit4:junit4] 1> 27794 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /get: solr.RealTimeGetHandler
[junit4:junit4] 1> 27794 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.RequestHandlers ? adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 1> 27794 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /replication: solr.ReplicationHandler
[junit4:junit4] 1> 27795 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.RequestHandlers ? created standard: solr.StandardRequestHandler
[junit4:junit4] 1> 27795 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /update: solr.UpdateRequestHandler
[junit4:junit4] 1> 27796 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 1> 27796 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /admin/ping: solr.PingRequestHandler
[junit4:junit4] 1> 27800 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.handler.loader.XMLLoader ? xsltCacheLifetimeSeconds=60
[junit4:junit4] 1> 27804 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.search.SolrIndexSearcher ? Opening Searcher@33ffa738 main
[junit4:junit4] 1> 27804 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.update.UpdateHandler ? Clearing tlog files, tlogDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty3/tlog
[junit4:junit4] 1> 27805 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.SolrCore ? Hard AutoCommit: disabled
[junit4:junit4] 1> 27806 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.SolrCore ? Soft AutoCommit: disabled
[junit4:junit4] 1> 27808 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.core.CoreContainer ? registering core: collection1
[junit4:junit4] 1> 27808 [searcherExecutor-57-thread-1] INFO org.apache.solr.core.SolrCore ? [collection1] Registered new searcher Searcher@33ffa738 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 1> 27808 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.cloud.ZkController ? Register replica - core:collection1 address:http://127.0.0.1:37922 collection:collection1 shard:shard1
[junit4:junit4] 1> 27814 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.cloud.ZkController ? We are http://127.0.0.1:37922/collection1/ and leader is http://127.0.0.1:38962/collection1/
[junit4:junit4] 1> 27815 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.cloud.ZkController ? No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37922
[junit4:junit4] 1> 27815 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.cloud.ZkController ? Core needs to recover:collection1
[junit4:junit4] 1> 27816 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.update.DefaultSolrCoreState ? Running recovery - first canceling any ongoing recovery
[junit4:junit4] 1> 27819 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 1> 27820 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? ###### startupVersions=[]
[junit4:junit4] 1> 27820 [coreLoadExecutor-56-thread-1] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 27821 [RecoveryThread] INFO org.apache.solr.cloud.ZkController ? publishing core=collection1 state=recovering
[junit4:junit4] 1> 27821 [RecoveryThread] INFO org.apache.solr.cloud.ZkController ? numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> 27823 [RecoveryThread] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> 27823 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.servlet.SolrDispatchFilter ? user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
[junit4:junit4] 1> 27825 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.servlet.SolrDispatchFilter ? SolrDispatchFilter.init() done
[junit4:junit4] 1> 27825 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> 27840 [qtp664387537-124] INFO org.apache.solr.handler.admin.CoreAdminHandler ? Going to wait for coreNodeName: 127.0.0.1:37922__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 1> 27870 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.eclipse.jetty.server.Server ? jetty-8.1.10.v20130312
[junit4:junit4] 1> 27873 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.eclipse.jetty.server.AbstractConnector ? Started SelectChannelConnector@127.0.0.1:49155
[junit4:junit4] 1> 27874 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.servlet.SolrDispatchFilter ? SolrDispatchFilter.init()
[junit4:junit4] 1> 27874 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrResourceLoader ? JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 1> 27874 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrResourceLoader ? using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty4-1371492024602
[junit4:junit4] 1> 27875 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty4-1371492024602/solr.xml
[junit4:junit4] 1> 27875 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? New CoreContainer 18591948
[junit4:junit4] 1> 27876 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty4-1371492024602/'
[junit4:junit4] 1> 27876 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrResourceLoader ? new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty4-1371492024602/'
[junit4:junit4] 1> 27937 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting socketTimeout to: 0
[junit4:junit4] 1> 27938 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting urlScheme to: http://
[junit4:junit4] 1> 27938 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting connTimeout to: 0
[junit4:junit4] 1> 27938 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting maxConnectionsPerHost to: 20
[junit4:junit4] 1> 27939 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting corePoolSize to: 0
[junit4:junit4] 1> 27939 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting maximumPoolSize to: 2147483647
[junit4:junit4] 1> 27940 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting maxThreadIdleTime to: 5
[junit4:junit4] 1> 27940 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting sizeOfQueue to: -1
[junit4:junit4] 1> 27940 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.handler.component.HttpShardHandlerFactory ? Setting fairnessPolicy to: false
[junit4:junit4] 1> 27941 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4] 1> 27944 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.logging.LogWatcher ? Registering Log Listener
[junit4:junit4] 1> 27945 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.ZkContainer ? Zookeeper client=127.0.0.1:58974/solr
[junit4:junit4] 1> 27945 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkController ? zkHost includes chroot
[junit4:junit4] 1> 27946 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> 27948 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager ? Watcher org.apache.solr.common.cloud.ConnectionManager@76b12cbe name:ZooKeeperConnection Watcher:127.0.0.1:58974 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> 27949 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Client is connected to ZooKeeper
[junit4:junit4] 1> 27951 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=0&connTimeout=0
[junit4:junit4] 1> 27955 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> 27956 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager ? Watcher org.apache.solr.common.cloud.ConnectionManager@48c54321 name:ZooKeeperConnection Watcher:127.0.0.1:58974/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> 27957 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ConnectionManager ? Client is connected to ZooKeeper
[junit4:junit4] 1> 27963 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cluster state from ZooKeeper...
[junit4:junit4] 1> 28120 [Thread-69] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 28122 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"127.0.0.1:37922__collection1",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"recovering",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:37922_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:37922"}
[junit4:junit4] 1> 28146 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 28147 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 28147 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 28147 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 28146 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 28146 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 28841 [qtp664387537-124] INFO org.apache.solr.handler.admin.CoreAdminHandler ? Waited coreNodeName: 127.0.0.1:37922__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 1> 28842 [qtp664387537-124] INFO org.apache.solr.servlet.SolrDispatchFilter ? [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:37922__collection1&state=recovering&nodeName=127.0.0.1:37922_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1004
[junit4:junit4] 1> 28967 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkController ? Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49155_
[junit4:junit4] 1> 28970 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /live_nodes/127.0.0.1:49155_
[junit4:junit4] 1> 28974 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 28974 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 28974 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 28974 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 28975 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (5)
[junit4:junit4] 1> 28974 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (5)
[junit4:junit4] 1> 28977 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 28977 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 28979 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (5)
[junit4:junit4] 1> 28979 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (5)
[junit4:junit4] 1> 28979 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (5)
[junit4:junit4] 1> 28980 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (5)
[junit4:junit4] 1> 28987 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.cloud.ZkController ? publishing core=collection1 state=down
[junit4:junit4] 1> 28987 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.cloud.ZkController ? numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> 29654 [Thread-69] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 29655 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":null,
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":null,
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:49155_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:49155"}
[junit4:junit4] 1> 29656 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Collection already exists with numShards=2
[junit4:junit4] 1> 29656 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Assigning new node to shard=shard2
[junit4:junit4] 1> 29662 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 29662 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 29662 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 29662 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 29662 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 29663 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 29990 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.CoreContainer ? Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty4-1371492024602/collection1
[junit4:junit4] 1> 29990 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.cloud.ZkController ? Check for collection zkNode:collection1
[junit4:junit4] 1> 29992 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.cloud.ZkController ? Collection zkNode exists
[junit4:junit4] 1> 29992 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.cloud.ZkController ? Load collection config from:/collections/collection1
[junit4:junit4] 1> 29994 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.SolrResourceLoader ? new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty4-1371492024602/collection1/'
[junit4:junit4] 1> 30025 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.SolrConfig ? Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 1> 30056 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.Config ? Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 1> 30058 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.schema.IndexSchema ? Reading Solr Schema from schema.xml
[junit4:junit4] 1> 30070 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.schema.IndexSchema ? [collection1] Schema name=test
[junit4:junit4] 1> 30401 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.schema.IndexSchema ? default search field in schema is text
[junit4:junit4] 1> 30403 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.schema.IndexSchema ? unique key field: id
[junit4:junit4] 1> 30405 [coreLoadExecutor-66-thread-1] WARN org.apache.solr.schema.IndexSchema ? Field text is not multivalued and destination for multiple copyFields (3)
[junit4:junit4] 1> 30405 [coreLoadExecutor-66-thread-1] WARN org.apache.solr.schema.IndexSchema ? Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
[junit4:junit4] 1> 30406 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.SolrCore ? org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> 30407 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.SolrCore ? [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./org.apache.solr.client.solrj.impl.CloudSolrServerTest-jetty4-1371492024602/collection1/, dataDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty4/
[junit4:junit4] 1> 30407 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.SolrCore ? JMX monitoring not detected for core: collection1
[junit4:junit4] 1> 30407 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.SolrCore ? org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> 30408 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.CachingDirectoryFactory ? return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty4
[junit4:junit4] 1> 30408 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.SolrCore ? New index directory detected: old=null new=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty4/index/
[junit4:junit4] 1> 30408 [coreLoadExecutor-66-thread-1] WARN org.apache.solr.core.SolrCore ? [collection1] Solr index directory './org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 1> 30409 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.CachingDirectoryFactory ? return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty4/index
[junit4:junit4] 1> 30411 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a817951 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2230658c),segFN=segments_1,generation=1}
[junit4:junit4] 1> 30411 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.SolrCore ? newest commit generation = 1
[junit4:junit4] 1> 30412 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /get: solr.RealTimeGetHandler
[junit4:junit4] 1> 30412 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.RequestHandlers ? adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 1> 30412 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /replication: solr.ReplicationHandler
[junit4:junit4] 1> 30413 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.RequestHandlers ? created standard: solr.StandardRequestHandler
[junit4:junit4] 1> 30413 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /update: solr.UpdateRequestHandler
[junit4:junit4] 1> 30413 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 1> 30414 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.RequestHandlers ? created /admin/ping: solr.PingRequestHandler
[junit4:junit4] 1> 30416 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.handler.loader.XMLLoader ? xsltCacheLifetimeSeconds=60
[junit4:junit4] 1> 30419 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.search.SolrIndexSearcher ? Opening Searcher@512aaf5 main
[junit4:junit4] 1> 30419 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.update.UpdateHandler ? Clearing tlog files, tlogDir=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty4/tlog
[junit4:junit4] 1> 30439 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.SolrCore ? Hard AutoCommit: disabled
[junit4:junit4] 1> 30440 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.SolrCore ? Soft AutoCommit: disabled
[junit4:junit4] 1> 30447 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.core.CoreContainer ? registering core: collection1
[junit4:junit4] 1> 30447 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.cloud.ZkController ? Register replica - core:collection1 address:http://127.0.0.1:49155 collection:collection1 shard:shard2
[junit4:junit4] 1> 30447 [searcherExecutor-67-thread-1] INFO org.apache.solr.core.SolrCore ? [collection1] Registered new searcher Searcher@512aaf5 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 1> 30454 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.cloud.ZkController ? We are http://127.0.0.1:49155/collection1/ and leader is http://127.0.0.1:59478/collection1/
[junit4:junit4] 1> 30455 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.cloud.ZkController ? No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49155
[junit4:junit4] 1> 30455 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.cloud.ZkController ? Core needs to recover:collection1
[junit4:junit4] 1> 30456 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.update.DefaultSolrCoreState ? Running recovery - first canceling any ongoing recovery
[junit4:junit4] 1> 30457 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 1> 30458 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? ###### startupVersions=[]
[junit4:junit4] 1> 30459 [coreLoadExecutor-66-thread-1] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 30459 [RecoveryThread] INFO org.apache.solr.cloud.ZkController ? publishing core=collection1 state=recovering
[junit4:junit4] 1> 30460 [RecoveryThread] INFO org.apache.solr.cloud.ZkController ? numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> 30462 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.servlet.SolrDispatchFilter ? user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0
[junit4:junit4] 1> 30463 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.servlet.SolrDispatchFilter ? SolrDispatchFilter.init() done
[junit4:junit4] 1> 30463 [RecoveryThread] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> 30464 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> 30472 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 30474 [qtp46299507-140] INFO org.apache.solr.handler.admin.CoreAdminHandler ? Going to wait for coreNodeName: 127.0.0.1:49155__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 1> 30477 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.AbstractFullDistribZkTestBase ? Wait for recoveries to finish - wait 30 for each attempt
[junit4:junit4] 1> 30477 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.BaseDistributedSearchTestCase ? Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
[junit4:junit4] 1> 30479 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 30843 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? Attempting to PeerSync from http://127.0.0.1:38962/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 1> 30846 [RecoveryThread] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 1> 30852 [RecoveryThread] INFO org.apache.solr.update.PeerSync ? PeerSync: core=collection1 url=http://127.0.0.1:37922 START replicas=[http://127.0.0.1:38962/collection1/] nUpdates=100
[junit4:junit4] 1> 30857 [RecoveryThread] WARN org.apache.solr.update.PeerSync ? no frame of reference to tell of we've missed updates
[junit4:junit4] 1> 30858 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 1> 30859 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? Starting Replication Recovery. core=collection1
[junit4:junit4] 1> 30859 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? Begin buffering updates. core=collection1
[junit4:junit4] 1> 30860 [RecoveryThread] INFO org.apache.solr.update.UpdateLog ? Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 1> 30860 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? Attempting to replicate from http://127.0.0.1:38962/collection1/. core=collection1
[junit4:junit4] 1> 30861 [RecoveryThread] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> 30873 [qtp664387537-123] INFO org.apache.solr.core.SolrCore ? [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 1> 30889 [qtp664387537-126] INFO org.apache.solr.update.UpdateHandler ? start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> 30892 [qtp664387537-126] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@b836456 lockFactory=org.apache.lucene.store.NativeFSLockFactory@bf6e58c),segFN=segments_1,generation=1}
[junit4:junit4] 1> 30892 [qtp664387537-126] INFO org.apache.solr.core.SolrCore ? newest commit generation = 1
[junit4:junit4] 1> 30893 [qtp664387537-126] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@b836456 lockFactory=org.apache.lucene.store.NativeFSLockFactory@bf6e58c),segFN=segments_1,generation=1}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@b836456 lockFactory=org.apache.lucene.store.NativeFSLockFactory@bf6e58c),segFN=segments_2,generation=2}
[junit4:junit4] 1> 30893 [qtp664387537-126] INFO org.apache.solr.core.SolrCore ? newest commit generation = 2
[junit4:junit4] 1> 30894 [qtp664387537-126] INFO org.apache.solr.search.SolrIndexSearcher ? Opening Searcher@322885a2 realtime
[junit4:junit4] 1> 30894 [qtp664387537-126] INFO org.apache.solr.update.UpdateHandler ? end_commit_flush
[junit4:junit4] 1> 30895 [qtp664387537-126] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 23
[junit4:junit4] 1> 30896 [RecoveryThread] INFO org.apache.solr.handler.ReplicationHandler ? Commits will be reserved for 10000
[junit4:junit4] 1> 30897 [RecoveryThread] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4] 1> 30906 [RecoveryThread] INFO org.apache.solr.handler.SnapPuller ? No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 1> 30926 [qtp664387537-125] INFO org.apache.solr.handler.ReplicationHandler ? Commits will be reserved for 10000
[junit4:junit4] 1> 30927 [qtp664387537-125] INFO org.apache.solr.core.SolrCore ? [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=18
[junit4:junit4] 1> 30929 [RecoveryThread] INFO org.apache.solr.handler.SnapPuller ? Master's generation: 2
[junit4:junit4] 1> 30929 [RecoveryThread] INFO org.apache.solr.handler.SnapPuller ? Slave's generation: 1
[junit4:junit4] 1> 30929 [RecoveryThread] INFO org.apache.solr.handler.SnapPuller ? Starting replication process
[junit4:junit4] 1> 30932 [qtp664387537-123] INFO org.apache.solr.core.SolrCore ? [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 1> 30933 [RecoveryThread] INFO org.apache.solr.handler.SnapPuller ? Number of files in latest index in master: 1
[junit4:junit4] 1> 30934 [RecoveryThread] INFO org.apache.solr.core.CachingDirectoryFactory ? return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty3/index.20130617190027706
[junit4:junit4] 1> 30935 [RecoveryThread] INFO org.apache.solr.handler.SnapPuller ? Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@73dffc71 lockFactory=org.apache.lucene.store.NativeFSLockFactory@62ce2bd2) fullCopy=false
[junit4:junit4] 1> 30940 [qtp664387537-126] INFO org.apache.solr.core.SolrCore ? [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=2
[junit4:junit4] 1> 30942 [RecoveryThread] INFO org.apache.solr.handler.SnapPuller ? Total time taken for download : 0 secs
[junit4:junit4] 1> 30947 [RecoveryThread] INFO org.apache.solr.update.DefaultSolrCoreState ? Creating new IndexWriter...
[junit4:junit4] 1> 30947 [RecoveryThread] INFO org.apache.solr.update.DefaultSolrCoreState ? Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 1> 30949 [RecoveryThread] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4c8796c7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a5104f6),segFN=segments_1,generation=1}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4c8796c7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a5104f6),segFN=segments_2,generation=2}
[junit4:junit4] 1> 30949 [RecoveryThread] INFO org.apache.solr.core.SolrCore ? newest commit generation = 2
[junit4:junit4] 1> 30950 [RecoveryThread] INFO org.apache.solr.update.DefaultSolrCoreState ? New IndexWriter is ready to be used.
[junit4:junit4] 1> 30950 [RecoveryThread] INFO org.apache.solr.search.SolrIndexSearcher ? Opening Searcher@7f640fd3 main
[junit4:junit4] 1> 30951 [searcherExecutor-57-thread-1] INFO org.apache.solr.core.SolrCore ? [collection1] Registered new searcher Searcher@7f640fd3 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> 30951 [RecoveryThread] INFO org.apache.solr.core.CachingDirectoryFactory ? looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty3/index.20130617190027706 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty3/index.20130617190027706;done=true>>]
[junit4:junit4] 1> 30951 [RecoveryThread] INFO org.apache.solr.core.CachingDirectoryFactory ? Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty3/index.20130617190027706
[junit4:junit4] 1> 30952 [RecoveryThread] INFO org.apache.solr.core.CachingDirectoryFactory ? Removing directory before core close: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty3/index.20130617190027706
[junit4:junit4] 1> 30952 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? No replay needed. core=collection1
[junit4:junit4] 1> 30952 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 1> 30952 [RecoveryThread] INFO org.apache.solr.cloud.ZkController ? publishing core=collection1 state=active
[junit4:junit4] 1> 30953 [RecoveryThread] INFO org.apache.solr.cloud.ZkController ? numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> 30954 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? Finished recovery process. core=collection1
[junit4:junit4] 1> 31171 [Thread-69] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 31172 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"127.0.0.1:49155__collection1",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"recovering",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:49155_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:49155"}
[junit4:junit4] 1> 31177 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"127.0.0.1:37922__collection1",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:37922_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:37922"}
[junit4:junit4] 1> 31191 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 31191 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 31191 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 31191 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 31191 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 31192 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 31475 [qtp46299507-140] INFO org.apache.solr.handler.admin.CoreAdminHandler ? Waited coreNodeName: 127.0.0.1:49155__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 1> 31476 [qtp46299507-140] INFO org.apache.solr.servlet.SolrDispatchFilter ? [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:49155__collection1&state=recovering&nodeName=127.0.0.1:49155_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4] 1> 31483 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 32486 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 33477 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? Attempting to PeerSync from http://127.0.0.1:59478/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 1> 33477 [RecoveryThread] INFO org.apache.solr.update.PeerSync ? PeerSync: core=collection1 url=http://127.0.0.1:49155 START replicas=[http://127.0.0.1:59478/collection1/] nUpdates=100
[junit4:junit4] 1> 33478 [RecoveryThread] WARN org.apache.solr.update.PeerSync ? no frame of reference to tell of we've missed updates
[junit4:junit4] 1> 33478 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 1> 33478 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? Starting Replication Recovery. core=collection1
[junit4:junit4] 1> 33479 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? Begin buffering updates. core=collection1
[junit4:junit4] 1> 33479 [RecoveryThread] INFO org.apache.solr.update.UpdateLog ? Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 1> 33480 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? Attempting to replicate from http://127.0.0.1:59478/collection1/. core=collection1
[junit4:junit4] 1> 33481 [RecoveryThread] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> 33486 [qtp46299507-140] INFO org.apache.solr.core.SolrCore ? [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 1> 33486 [qtp46299507-139] INFO org.apache.solr.update.UpdateHandler ? start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> 33488 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 33488 [qtp46299507-139] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7e1b4e9d lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f7b8f60),segFN=segments_1,generation=1}
[junit4:junit4] 1> 33489 [qtp46299507-139] INFO org.apache.solr.core.SolrCore ? newest commit generation = 1
[junit4:junit4] 1> 33490 [qtp46299507-139] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7e1b4e9d lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f7b8f60),segFN=segments_1,generation=1}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7e1b4e9d lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f7b8f60),segFN=segments_2,generation=2}
[junit4:junit4] 1> 33490 [qtp46299507-139] INFO org.apache.solr.core.SolrCore ? newest commit generation = 2
[junit4:junit4] 1> 33491 [qtp46299507-139] INFO org.apache.solr.search.SolrIndexSearcher ? Opening Searcher@7a8de550 realtime
[junit4:junit4] 1> 33491 [qtp46299507-139] INFO org.apache.solr.update.UpdateHandler ? end_commit_flush
[junit4:junit4] 1> 33491 [qtp46299507-139] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 1> 33492 [RecoveryThread] INFO org.apache.solr.handler.ReplicationHandler ? Commits will be reserved for 10000
[junit4:junit4] 1> 33493 [RecoveryThread] INFO org.apache.solr.handler.SnapPuller ? No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 1> 33496 [qtp46299507-142] INFO org.apache.solr.handler.ReplicationHandler ? Commits will be reserved for 10000
[junit4:junit4] 1> 33496 [qtp46299507-142] INFO org.apache.solr.core.SolrCore ? [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 1> 33497 [RecoveryThread] INFO org.apache.solr.handler.SnapPuller ? Master's generation: 2
[junit4:junit4] 1> 33497 [RecoveryThread] INFO org.apache.solr.handler.SnapPuller ? Slave's generation: 1
[junit4:junit4] 1> 33498 [RecoveryThread] INFO org.apache.solr.handler.SnapPuller ? Starting replication process
[junit4:junit4] 1> 33500 [qtp46299507-140] INFO org.apache.solr.core.SolrCore ? [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 1> 33501 [RecoveryThread] INFO org.apache.solr.handler.SnapPuller ? Number of files in latest index in master: 1
[junit4:junit4] 1> 33502 [RecoveryThread] INFO org.apache.solr.core.CachingDirectoryFactory ? return new directory for ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty4/index.20130617190030273
[junit4:junit4] 1> 33502 [RecoveryThread] INFO org.apache.solr.handler.SnapPuller ? Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@4b998b66 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5dd9559e) fullCopy=false
[junit4:junit4] 1> 33505 [qtp46299507-139] INFO org.apache.solr.core.SolrCore ? [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 1> 33506 [RecoveryThread] INFO org.apache.solr.handler.SnapPuller ? Total time taken for download : 0 secs
[junit4:junit4] 1> 33507 [RecoveryThread] INFO org.apache.solr.update.DefaultSolrCoreState ? Creating new IndexWriter...
[junit4:junit4] 1> 33507 [RecoveryThread] INFO org.apache.solr.update.DefaultSolrCoreState ? Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 1> 33509 [RecoveryThread] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a817951 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2230658c),segFN=segments_1,generation=1}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a817951 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2230658c),segFN=segments_2,generation=2}
[junit4:junit4] 1> 33509 [RecoveryThread] INFO org.apache.solr.core.SolrCore ? newest commit generation = 2
[junit4:junit4] 1> 33509 [RecoveryThread] INFO org.apache.solr.update.DefaultSolrCoreState ? New IndexWriter is ready to be used.
[junit4:junit4] 1> 33510 [RecoveryThread] INFO org.apache.solr.search.SolrIndexSearcher ? Opening Searcher@1ec375d4 main
[junit4:junit4] 1> 33510 [searcherExecutor-67-thread-1] INFO org.apache.solr.core.SolrCore ? [collection1] Registered new searcher Searcher@1ec375d4 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> 33511 [RecoveryThread] INFO org.apache.solr.core.CachingDirectoryFactory ? looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty4/index.20130617190030273 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty4/index.20130617190030273;done=true>>]
[junit4:junit4] 1> 33511 [RecoveryThread] INFO org.apache.solr.core.CachingDirectoryFactory ? Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty4/index.20130617190030273
[junit4:junit4] 1> 33511 [RecoveryThread] INFO org.apache.solr.core.CachingDirectoryFactory ? Removing directory before core close: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty4/index.20130617190030273
[junit4:junit4] 1> 33511 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? No replay needed. core=collection1
[junit4:junit4] 1> 33512 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 1> 33512 [RecoveryThread] INFO org.apache.solr.cloud.ZkController ? publishing core=collection1 state=active
[junit4:junit4] 1> 33512 [RecoveryThread] INFO org.apache.solr.cloud.ZkController ? numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> 33514 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy ? Finished recovery process. core=collection1
[junit4:junit4] 1> 34204 [Thread-69] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 34205 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"127.0.0.1:49155__collection1",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:49155_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:49155"}
[junit4:junit4] 1> 34301 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 34302 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 34302 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 34302 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 34302 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 34301 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 34491 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 34492 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.BaseDistributedSearchTestCase ? Recoveries finished - collection: collection1
[junit4:junit4] 1> 34498 [qtp1018857458-106] INFO org.apache.solr.update.UpdateHandler ? start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> 34501 [qtp1018857458-106] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7bfea51e lockFactory=org.apache.lucene.store.NativeFSLockFactory@57ed07c5),segFN=segments_1,generation=1}
[junit4:junit4] 1> 34502 [qtp1018857458-106] INFO org.apache.solr.core.SolrCore ? newest commit generation = 1
[junit4:junit4] 1> 34502 [qtp1018857458-106] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7bfea51e lockFactory=org.apache.lucene.store.NativeFSLockFactory@57ed07c5),segFN=segments_1,generation=1}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7bfea51e lockFactory=org.apache.lucene.store.NativeFSLockFactory@57ed07c5),segFN=segments_2,generation=2}
[junit4:junit4] 1> 34503 [qtp1018857458-106] INFO org.apache.solr.core.SolrCore ? newest commit generation = 2
[junit4:junit4] 1> 34503 [qtp1018857458-106] INFO org.apache.solr.search.SolrIndexSearcher ? Opening Searcher@2e806493 main
[junit4:junit4] 1> 34504 [qtp1018857458-106] INFO org.apache.solr.update.UpdateHandler ? end_commit_flush
[junit4:junit4] 1> 34504 [searcherExecutor-27-thread-1] INFO org.apache.solr.core.SolrCore ? [collection1] Registered new searcher Searcher@2e806493 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> 34504 [qtp1018857458-106] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 1> 34512 [qtp664387537-123] INFO org.apache.solr.update.SolrCmdDistributor ? Distrib commit to:[StdNode: http://127.0.0.1:38962/collection1/, StdNode: http://127.0.0.1:37922/collection1/, StdNode: http://127.0.0.1:59478/collection1/, StdNode: http://127.0.0.1:49155/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 1> 34518 [qtp664387537-126] INFO org.apache.solr.update.UpdateHandler ? start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> 34518 [qtp46299507-142] INFO org.apache.solr.update.UpdateHandler ? start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> 34518 [qtp664387537-126] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@b836456 lockFactory=org.apache.lucene.store.NativeFSLockFactory@bf6e58c),segFN=segments_2,generation=2}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@b836456 lockFactory=org.apache.lucene.store.NativeFSLockFactory@bf6e58c),segFN=segments_3,generation=3}
[junit4:junit4] 1> 34519 [qtp664387537-126] INFO org.apache.solr.core.SolrCore ? newest commit generation = 3
[junit4:junit4] 1> 34519 [qtp664387537-126] INFO org.apache.solr.search.SolrIndexSearcher ? Opening Searcher@34e68412 main
[junit4:junit4] 1> 34520 [qtp664387537-126] INFO org.apache.solr.update.UpdateHandler ? end_commit_flush
[junit4:junit4] 1> 34520 [searcherExecutor-37-thread-1] INFO org.apache.solr.core.SolrCore ? [collection1] Registered new searcher Searcher@34e68412 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> 34522 [qtp664387537-126] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 1> 34522 [qtp631060060-155] INFO org.apache.solr.update.UpdateHandler ? start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> 34524 [qtp46299507-142] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7e1b4e9d lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f7b8f60),segFN=segments_2,generation=2}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7e1b4e9d lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f7b8f60),segFN=segments_3,generation=3}
[junit4:junit4] 1> 34523 [qtp529058223-175] INFO org.apache.solr.update.UpdateHandler ? start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> 34526 [qtp46299507-142] INFO org.apache.solr.core.SolrCore ? newest commit generation = 3
[junit4:junit4] 1> 34526 [qtp631060060-155] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4c8796c7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a5104f6),segFN=segments_2,generation=2}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4c8796c7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a5104f6),segFN=segments_3,generation=3}
[junit4:junit4] 1> 34528 [qtp631060060-155] INFO org.apache.solr.core.SolrCore ? newest commit generation = 3
[junit4:junit4] 1> 34528 [qtp46299507-142] INFO org.apache.solr.search.SolrIndexSearcher ? Opening Searcher@68df2647 main
[junit4:junit4] 1> 34527 [qtp529058223-175] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a817951 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2230658c),segFN=segments_2,generation=2}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a817951 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2230658c),segFN=segments_3,generation=3}
[junit4:junit4] 1> 34529 [qtp529058223-175] INFO org.apache.solr.core.SolrCore ? newest commit generation = 3
[junit4:junit4] 1> 34529 [searcherExecutor-47-thread-1] INFO org.apache.solr.core.SolrCore ? [collection1] Registered new searcher Searcher@68df2647 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> 34529 [qtp46299507-142] INFO org.apache.solr.update.UpdateHandler ? end_commit_flush
[junit4:junit4] 1> 34529 [qtp631060060-155] INFO org.apache.solr.search.SolrIndexSearcher ? Opening Searcher@2d24792a main
[junit4:junit4] 1> 34531 [qtp46299507-142] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 14
[junit4:junit4] 1> 34530 [qtp529058223-175] INFO org.apache.solr.search.SolrIndexSearcher ? Opening Searcher@70397660 main
[junit4:junit4] 1> 34531 [searcherExecutor-57-thread-1] INFO org.apache.solr.core.SolrCore ? [collection1] Registered new searcher Searcher@2d24792a main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> 34531 [qtp631060060-155] INFO org.apache.solr.update.UpdateHandler ? end_commit_flush
[junit4:junit4] 1> 34532 [searcherExecutor-67-thread-1] INFO org.apache.solr.core.SolrCore ? [collection1] Registered new searcher Searcher@70397660 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> 34532 [qtp529058223-175] INFO org.apache.solr.update.UpdateHandler ? end_commit_flush
[junit4:junit4] 1> 34533 [qtp631060060-155] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4] 1> 34534 [qtp529058223-175] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4] 1> 34535 [qtp664387537-123] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 24
[junit4:junit4] 1> 34537 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 34541 [qtp664387537-125] INFO org.apache.solr.core.SolrCore ? [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 1> 34546 [qtp631060060-156] INFO org.apache.solr.core.SolrCore ? [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 1> 34550 [qtp46299507-140] INFO org.apache.solr.core.SolrCore ? [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 1> 34554 [qtp529058223-172] INFO org.apache.solr.core.SolrCore ? [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 1> 36560 [qtp1018857458-105] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1438113630342086656)} 0 2
[junit4:junit4] 1> 36571 [qtp529058223-172] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1438113630350475264&update.from=http://127.0.0.1:59478/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438113630350475264)} 0 1
[junit4:junit4] 1> 36571 [qtp631060060-156] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1438113630350475264&update.from=http://127.0.0.1:38962/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438113630350475264)} 0 1
[junit4:junit4] 1> 36572 [qtp46299507-140] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1438113630350475264)} 0 5
[junit4:junit4] 1> 36572 [qtp664387537-125] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1438113630350475264)} 0 6
[junit4:junit4] 1> 36573 [qtp631060060-157] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 10
[junit4:junit4] 1> 36582 [qtp1018857458-106] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1438113630360961024)]} 0 6
[junit4:junit4] 1> 36594 [qtp631060060-155] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:38962/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438113630373543936)]} 0 1
[junit4:junit4] 1> 36595 [qtp664387537-124] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:59478/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1438113630373543936)]} 0 6
[junit4:junit4] 1> 36596 [qtp46299507-141] INFO org.apache.solr.update.processor.LogUpdateProcessor ? [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0]} 0 12
[junit4:junit4] 1> 36598 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ChaosMonkey ? monkey: stop shard! 47252
[junit4:junit4] 1> 36599 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? Shutting down CoreContainer instance=276280784
[junit4:junit4] 1> 37314 [Thread-69] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 37316 [Thread-69] INFO org.apache.solr.cloud.Overseer ? Update state numShards=null message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"127.0.0.1:47252__collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "node_name":"127.0.0.1:47252_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:47252"}
[junit4:junit4] 1> 37321 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 37321 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 37321 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 37321 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 37321 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 38600 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkController ? publishing core=collection1 state=down
[junit4:junit4] 1> 38601 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkController ? numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> 38603 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrCore ? [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@79c68933
[junit4:junit4] 1> 38603 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.update.UpdateHandler ? closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=1,errors=0,cumulative_adds=1,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 1> 38604 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.update.SolrCoreState ? Closing SolrCoreState
[junit4:junit4] 1> 38604 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.update.DefaultSolrCoreState ? SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 1> 38605 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.update.DefaultSolrCoreState ? closing IndexWriter with IndexWriterCloser
[junit4:junit4] 1> 38629 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7bfea51e lockFactory=org.apache.lucene.store.NativeFSLockFactory@57ed07c5),segFN=segments_2,generation=2}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7bfea51e lockFactory=org.apache.lucene.store.NativeFSLockFactory@57ed07c5),segFN=segments_3,generation=3}
[junit4:junit4] 1> 38630 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrCore ? newest commit generation = 3
[junit4:junit4] 1> 38631 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrCore ? [collection1] Closing main searcher on request.
[junit4:junit4] 1> 38632 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CachingDirectoryFactory ? Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 1> 38632 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CachingDirectoryFactory ? looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/control/data [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/control/data;done=false>>]
[junit4:junit4] 1> 38632 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CachingDirectoryFactory ? Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/control/data
[junit4:junit4] 1> 38633 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CachingDirectoryFactory ? looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/control/data/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/control/data/index;done=false>>]
[junit4:junit4] 1> 38633 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CachingDirectoryFactory ? Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/control/data/index
[junit4:junit4] 1> 38635 [Thread-69] INFO org.apache.solr.cloud.Overseer ? According to ZK I (id=89882100497645571-127.0.0.1:47252_-n_0000000000) am no longer a leader.
[junit4:junit4] 1> 38637 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 38637 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 38637 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> 38638 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (4)
[junit4:junit4] 1> 38639 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 38638 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] WARN org.apache.solr.common.cloud.ZkStateReader ? ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 1> 38639 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] WARN org.apache.solr.common.cloud.ZkStateReader ? ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 1> 38639 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (4)
[junit4:junit4] 1> 38640 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 38641 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (4)
[junit4:junit4] 1> 38641 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (4)
[junit4:junit4] 1> 38643 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /overseer_elect/leader
[junit4:junit4] 1> 38646 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.cloud.Overseer ? Overseer (id=89882100497645574-127.0.0.1:38962_-n_0000000001) starting
[junit4:junit4] 1> 38660 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.eclipse.jetty.server.handler.ContextHandler ? stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 1> 38667 [Overseer-89882100497645574-127.0.0.1:38962_-n_0000000001] INFO org.apache.solr.cloud.OverseerCollectionProcessor ? Process current queue of collection creations
[junit4:junit4] 1> 38668 [Thread-113] INFO org.apache.solr.cloud.Overseer ? Starting to work on the main queue
[junit4:junit4] 1> 38671 [Thread-113] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 38672 [Thread-113] INFO org.apache.solr.cloud.Overseer ? Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"127.0.0.1:47252__collection1",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:47252_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:47252"}
[junit4:junit4] 1> 38677 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 38677 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 38677 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 38677 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 38713 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ChaosMonkey ? monkey: stop shard! 38962
[junit4:junit4] 1> 38713 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? Shutting down CoreContainer instance=777887041
[junit4:junit4] 1> 39715 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkController ? publishing core=collection1 state=down
[junit4:junit4] 1> 39715 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkController ? numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> 39717 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrCore ? [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@68e1aabd
[junit4:junit4] 1> 39718 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.update.UpdateHandler ? closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=1,errors=0,cumulative_adds=1,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 1> 39718 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.update.SolrCoreState ? Closing SolrCoreState
[junit4:junit4] 1> 39719 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.update.DefaultSolrCoreState ? SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 1> 39719 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.update.DefaultSolrCoreState ? closing IndexWriter with IndexWriterCloser
[junit4:junit4] 1> 39724 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@b836456 lockFactory=org.apache.lucene.store.NativeFSLockFactory@bf6e58c),segFN=segments_2,generation=2}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@b836456 lockFactory=org.apache.lucene.store.NativeFSLockFactory@bf6e58c),segFN=segments_3,generation=3}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@b836456 lockFactory=org.apache.lucene.store.NativeFSLockFactory@bf6e58c),segFN=segments_4,generation=4}
[junit4:junit4] 1> 39724 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrCore ? newest commit generation = 4
[junit4:junit4] 1> 39726 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrCore ? [collection1] Closing main searcher on request.
[junit4:junit4] 1> 39726 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CachingDirectoryFactory ? Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 1> 39727 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CachingDirectoryFactory ? looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty1 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty1;done=false>>]
[junit4:junit4] 1> 39727 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CachingDirectoryFactory ? Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty1
[junit4:junit4] 1> 39728 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CachingDirectoryFactory ? looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty1/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty1/index;done=false>>]
[junit4:junit4] 1> 39728 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CachingDirectoryFactory ? Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty1/index
[junit4:junit4] 1> 39729 [Thread-113] INFO org.apache.solr.cloud.Overseer ? According to ZK I (id=89882100497645574-127.0.0.1:38962_-n_0000000001) am no longer a leader.
[junit4:junit4] 1> 39731 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (3)
[junit4:junit4] 1> 39731 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> 39732 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /overseer_elect/leader
[junit4:junit4] 1> 39732 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.cloud.ShardLeaderElectionContext ? Running the leader process.
[junit4:junit4] 1> 39732 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> 39732 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] WARN org.apache.solr.common.cloud.ZkStateReader ? ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 1> 39734 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] WARN org.apache.solr.common.cloud.ZkStateReader ? ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 1> 39736 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.cloud.ShardLeaderElectionContext ? Checking if I should try and be the leader.
[junit4:junit4] 1> 39736 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.cloud.Overseer ? Overseer (id=89882100497645576-127.0.0.1:59478_-n_0000000002) starting
[junit4:junit4] 1> 39736 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.cloud.ShardLeaderElectionContext ? My last published State was Active, it's okay to be the leader.
[junit4:junit4] 1> 39737 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.cloud.ShardLeaderElectionContext ? I may be the new leader - try and sync
[junit4:junit4] 1> 39737 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.cloud.SyncStrategy ? Sync replicas to http://127.0.0.1:37922/collection1/
[junit4:junit4] 1> 39738 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.update.PeerSync ? PeerSync: core=collection1 url=http://127.0.0.1:37922 START replicas=[http://127.0.0.1:38962/collection1/] nUpdates=100
[junit4:junit4] 1> 39743 [Overseer-89882100497645576-127.0.0.1:59478_-n_0000000002] INFO org.apache.solr.cloud.OverseerCollectionProcessor ? Process current queue of collection creations
[junit4:junit4] 1> 39744 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating live nodes... (3)
[junit4:junit4] 1> 39744 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> 39745 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] WARN org.apache.solr.update.PeerSync ? PeerSync: core=collection1 url=http://127.0.0.1:37922 couldn't connect to http://127.0.0.1:38962/collection1/, counting as success
[junit4:junit4] 1> 39745 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.eclipse.jetty.server.handler.ContextHandler ? stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 1> 39745 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.update.PeerSync ? PeerSync: core=collection1 url=http://127.0.0.1:37922 DONE. sync succeeded
[junit4:junit4] 1> 39746 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.cloud.SyncStrategy ? Sync Success - now sync replicas to me
[junit4:junit4] 1> 39747 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.cloud.SyncStrategy ? http://127.0.0.1:37922/collection1/: try and ask http://127.0.0.1:38962/collection1/ to sync
[junit4:junit4] 1> 39748 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] ERROR org.apache.solr.cloud.SyncStrategy ? Sync request error: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:38962/collection1
[junit4:junit4] 1> 39749 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.cloud.SyncStrategy ? http://127.0.0.1:37922/collection1/: Sync failed - asking replica (http://127.0.0.1:38962/collection1/) to recover.
[junit4:junit4] 1> 39749 [Thread-114] INFO org.apache.solr.cloud.Overseer ? Starting to work on the main queue
[junit4:junit4] 1> 39751 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.cloud.ShardLeaderElectionContext ? I am the new leader: http://127.0.0.1:37922/collection1/
[junit4:junit4] 1> 39751 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.SolrZkClient ? makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 1> 39751 [recoveryCmdExecutor-61-thread-1] INFO org.apache.solr.client.solrj.impl.HttpClientUtil ? Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> 39755 [Thread-114] INFO org.apache.solr.common.cloud.ZkStateReader ? Updating cloud state from ZooKeeper...
[junit4:junit4] 1> 39757 [Thread-114] INFO org.apache.solr.cloud.Overseer ? Update state numShards=null message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"127.0.0.1:38962__collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:38962_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:38962"}
[junit4:junit4] 1> 39759 [recoveryCmdExecutor-61-thread-1] ERROR org.apache.solr.cloud.SyncStrategy ? http://127.0.0.1:37922/collection1/: Could not tell a replica to recover:org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:38962
[junit4:junit4] 1> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
[junit4:junit4] 1> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4] 1> at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:298)
[junit4:junit4] 1> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 1> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 1> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 1> Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:38962 refused
[junit4:junit4] 1> at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
[junit4:junit4] 1> at
[...truncated too long message...]
pdating... (live nodes size: 1)
[junit4:junit4] 1> 41971 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ChaosMonkey ? monkey: stop shard! 49155
[junit4:junit4] 1> 41971 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CoreContainer ? Shutting down CoreContainer instance=18591948
[junit4:junit4] 1> 42111 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] WARN org.apache.solr.common.cloud.ZkStateReader ? ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 1> 42112 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> 42112 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] WARN org.apache.solr.common.cloud.ZkStateReader ? ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 1> 42973 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkController ? publishing core=collection1 state=down
[junit4:junit4] 1> 42973 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkController ? numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> 42975 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrCore ? [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@672d82a2
[junit4:junit4] 1> 42975 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.update.UpdateHandler ? closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=1,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 1> 42976 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.update.SolrCoreState ? Closing SolrCoreState
[junit4:junit4] 1> 42976 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.update.DefaultSolrCoreState ? SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 1> 42977 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.update.DefaultSolrCoreState ? closing IndexWriter with IndexWriterCloser
[junit4:junit4] 1> 42977 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrCore ? SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a817951 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2230658c),segFN=segments_3,generation=3}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a817951 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2230658c),segFN=segments_4,generation=4}
[junit4:junit4] 1> 42978 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrCore ? newest commit generation = 4
[junit4:junit4] 1> 42979 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.SolrCore ? [collection1] Closing main searcher on request.
[junit4:junit4] 1> 42979 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CachingDirectoryFactory ? Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 1> 42980 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CachingDirectoryFactory ? looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty4;done=false>>]
[junit4:junit4] 1> 42980 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CachingDirectoryFactory ? Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty4
[junit4:junit4] 1> 42980 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CachingDirectoryFactory ? looking to close ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty4/index;done=false>>]
[junit4:junit4] 1> 42981 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.core.CachingDirectoryFactory ? Closing directory: ./org.apache.solr.client.solrj.impl.CloudSolrServerTest-1371492011934/jetty4/index
[junit4:junit4] 1> 42982 [Thread-118] INFO org.apache.solr.cloud.Overseer ? According to ZK I (id=89882100497645580-127.0.0.1:49155_-n_0000000004) am no longer a leader.
[junit4:junit4] 1> 43004 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.eclipse.jetty.server.handler.ContextHandler ? stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 1> 43060 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.SolrTestCaseJ4 ? ###Ending testDistribSearch
[junit4:junit4] 1> 43062 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]] INFO org.apache.solr.cloud.ZkTestServer ? connecting to 127.0.0.1:58974 58974
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-solrj/test/J0/./solrtest-CloudSolrServerTest-1371492011935
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=CloudSolrServerTest -Dtests.method=testDistribSearch -Dtests.seed=739BCB3D49E058B1 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=pl -Dtests.timezone=Africa/Bangui -Dtests.file.encoding=US-ASCII
[junit4:junit4] ERROR 27.9s J0 | CloudSolrServerTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.NullPointerException
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([739BCB3D49E058B1:F27D45253EBF388D]:0)
[junit4:junit4] > at org.apache.solr.client.solrj.impl.CloudSolrServerTest.doTest(CloudSolrServerTest.java:108)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 1> 43106 [SUITE-CloudSolrServerTest-seed#[739BCB3D49E058B1]-worker] INFO org.apache.solr.SolrTestCaseJ4 ? ###deleteCore
[junit4:junit4] 2> 27956 T87 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 1> 44283 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] WARN org.apache.solr.common.cloud.ZkStateReader ? ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 1> 44283 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] INFO org.apache.solr.common.cloud.ZkStateReader ? A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> 44284 [TEST-CloudSolrServerTest.testDistribSearch-seed#[739BCB3D49E058B1]-EventThread] WARN org.apache.solr.common.cloud.ZkStateReader ? ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {timestamp=PostingsFormat(name=Memory doPackFST= false), range_facet_l=PostingsFormat(name=Memory doPackFST= false), id=PostingsFormat(name=MockSep), range_facet_sl=PostingsFormat(name=SimpleText), a_t=PostingsFormat(name=Memory doPackFST= false), text=PostingsFormat(name=SimpleText), range_facet_si=MockFixedIntBlock(blockSize=1179), _version_=PostingsFormat(name=Memory doPackFST= false), multiDefault=MockFixedIntBlock(blockSize=1179), intDefault=PostingsFormat(name=Memory doPackFST= false)}, docValues:{}, sim=DefaultSimilarity, locale=pl, timezone=Africa/Bangui
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=8,threads=5,free=79306248,total=238813184
[junit4:junit4] 2> NOTE: All tests run in this JVM: [LargeVolumeJettyTest, LargeVolumeBinaryJettyTest, SolrExampleXMLTest, ContentStreamTest, TestUpdateRequestCodec, CloudSolrServerTest]
[junit4:junit4] Completed on J0 in 29.25s, 1 test, 1 error <<< FAILURES!
[...truncated 97 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:194: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 46 suites, 272 tests, 1 error
Total time: 37 minutes 59 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_21) - Build #
6182 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6182/
Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseSerialGC
1 tests failed.
FAILED: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
expected:<614> but was:<72>
Stack Trace:
java.lang.AssertionError: expected:<614> but was:<72>
at __randomizedtesting.SeedInfo.seed([B7A833C1E8811B6C:364EBDD99FDE7B50]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9295 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4] 2> 119271 T583 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /u_/
[junit4:junit4] 2> 119275 T583 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1371505385572
[junit4:junit4] 2> 119275 T583 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 119276 T584 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 119376 T583 oasc.ZkTestServer.run start zk server on port:34322
[junit4:junit4] 2> 119377 T583 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 119484 T590 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@81edaec name:ZooKeeperConnection Watcher:127.0.0.1:34322 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 119485 T583 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 119485 T583 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 119490 T583 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 119492 T592 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60d77020 name:ZooKeeperConnection Watcher:127.0.0.1:34322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 119492 T583 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 119492 T583 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 119495 T583 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 119498 T583 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 119500 T583 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 119504 T583 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 119504 T583 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 119509 T583 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 119509 T583 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 119513 T583 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 119513 T583 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 119516 T583 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 119516 T583 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 119519 T583 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 119519 T583 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 119523 T583 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 119523 T583 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 119526 T583 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 119526 T583 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 119529 T583 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 119530 T583 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 119532 T583 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 119533 T583 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 119606 T583 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 119608 T583 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34380
[junit4:junit4] 2> 119609 T583 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 119609 T583 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 119610 T583 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371505385833
[junit4:junit4] 2> 119610 T583 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371505385833/solr.xml
[junit4:junit4] 2> 119610 T583 oasc.CoreContainer.<init> New CoreContainer 2115155902
[junit4:junit4] 2> 119610 T583 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371505385833/'
[junit4:junit4] 2> 119611 T583 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371505385833/'
[junit4:junit4] 2> 119658 T583 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 119658 T583 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 119659 T583 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 119659 T583 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 119659 T583 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 119659 T583 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 119660 T583 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 119660 T583 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 119660 T583 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 119661 T583 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 119664 T583 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 119665 T583 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34322/solr
[junit4:junit4] 2> 119665 T583 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 119666 T583 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 119667 T603 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c2a0717 name:ZooKeeperConnection Watcher:127.0.0.1:34322 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 119667 T583 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 119669 T583 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 119672 T583 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 119673 T605 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6bcae731 name:ZooKeeperConnection Watcher:127.0.0.1:34322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 119673 T583 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 119675 T583 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 119678 T583 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 119680 T583 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 119682 T583 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34380_u_
[junit4:junit4] 2> 119683 T583 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34380_u_
[junit4:junit4] 2> 119688 T583 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 119695 T583 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 119697 T583 oasc.Overseer.start Overseer (id=89882976948977667-127.0.0.1:34380_u_-n_0000000000) starting
[junit4:junit4] 2> 119699 T583 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 119702 T607 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 119703 T583 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 119706 T583 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 119707 T583 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 119710 T606 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 119712 T608 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 119713 T608 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 121217 T606 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 121218 T606 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:34380_u_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:34380/u_"}
[junit4:junit4] 2> 121218 T606 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 121219 T606 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 121225 T605 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 121714 T608 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371505385833/collection1
[junit4:junit4] 2> 121714 T608 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 121715 T608 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 121715 T608 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 121717 T608 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371505385833/collection1/'
[junit4:junit4] 2> 121717 T608 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1371505385833/collection1/lib/README' to classloader
[junit4:junit4] 2> 121717 T608 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1371505385833/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 121754 T608 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 121793 T608 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 121794 T608 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 121798 T608 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 122123 T608 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 122129 T608 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 122131 T608 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 122146 T608 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 122149 T608 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 122152 T608 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 122153 T608 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 122153 T608 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 122153 T608 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 122154 T608 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 122154 T608 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 122154 T608 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 122155 T608 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1371505385833/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371505385572/control/data/
[junit4:junit4] 2> 122155 T608 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@25c22d44
[junit4:junit4] 2> 122155 T608 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 122156 T608 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371505385572/control/data
[junit4:junit4] 2> 122156 T608 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371505385572/control/data/index/
[junit4:junit4] 2> 122156 T608 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371505385572/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 122156 T608 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371505385572/control/data/index
[junit4:junit4] 2> 122158 T608 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60c8fab lockFactory=org.apache.lucene.store.NativeFSLockFactory@68fbc5d3),segFN=segments_1,generation=1}
[junit4:junit4] 2> 122158 T608 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 122159 T608 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 122159 T608 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 122160 T608 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 122160 T608 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 122161 T608 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 122161 T608 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 122161 T608 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 122161 T608 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 122162 T608 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 122164 T608 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 122166 T608 oass.SolrIndexSearcher.<init> Opening Searcher@1a58697d main
[junit4:junit4] 2> 122166 T608 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371505385572/control/data/tlog
[junit4:junit4] 2> 122166 T608 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 122167 T608 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 122170 T609 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1a58697d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 122171 T608 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 122171 T608 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:34380/u_ collection:control_collection shard:shard1
[junit4:junit4] 2> 122171 T608 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 122181 T608 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 122184 T608 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 122184 T608 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 122184 T608 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:34380/u_/collection1/
[junit4:junit4] 2> 122185 T608 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 122185 T608 oasc.SyncStrategy.syncToMe http://127.0.0.1:34380/u_/collection1/ has no replicas
[junit4:junit4] 2> 122185 T608 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:34380/u_/collection1/
[junit4:junit4] 2> 122185 T608 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 122730 T606 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 122738 T605 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 122792 T608 oasc.ZkController.register We are http://127.0.0.1:34380/u_/collection1/ and leader is http://127.0.0.1:34380/u_/collection1/
[junit4:junit4] 2> 122792 T608 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:34380/u_
[junit4:junit4] 2> 122792 T608 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 122792 T608 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 122793 T608 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 122795 T608 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 122796 T583 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 122796 T583 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 122796 T583 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 122801 T583 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 122802 T583 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 122804 T612 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a11d9c5 name:ZooKeeperConnection Watcher:127.0.0.1:34322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 122805 T583 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 122806 T583 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 122809 T583 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 122881 T583 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 122883 T583 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56556
[junit4:junit4] 2> 122883 T583 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 122884 T583 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 122884 T583 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371505389106
[junit4:junit4] 2> 122884 T583 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371505389106/solr.xml
[junit4:junit4] 2> 122884 T583 oasc.CoreContainer.<init> New CoreContainer 620847560
[junit4:junit4] 2> 122885 T583 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371505389106/'
[junit4:junit4] 2> 122885 T583 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371505389106/'
[junit4:junit4] 2> 122931 T583 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 122931 T583 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 122931 T583 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 122932 T583 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 122932 T583 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 122932 T583 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 122932 T583 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 122933 T583 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 122933 T583 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 122933 T583 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 122937 T583 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 122937 T583 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34322/solr
[junit4:junit4] 2> 122937 T583 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 122938 T583 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 122940 T623 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@299abca9 name:ZooKeeperConnection Watcher:127.0.0.1:34322 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 122940 T583 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 122942 T583 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 122945 T583 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 122947 T625 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2cd15469 name:ZooKeeperConnection Watcher:127.0.0.1:34322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 122947 T583 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 122951 T583 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 123955 T583 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56556_u_
[junit4:junit4] 2> 123956 T583 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56556_u_
[junit4:junit4] 2> 123963 T605 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 123964 T612 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 123964 T625 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 123965 T605 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 123969 T626 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 123969 T626 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 124244 T606 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 124245 T606 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:34380_u_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:34380/u_"}
[junit4:junit4] 2> 124250 T606 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56556_u_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56556/u_"}
[junit4:junit4] 2> 124251 T606 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 124251 T606 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 124258 T605 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 124258 T612 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 124258 T625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 124971 T626 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371505389106/collection1
[junit4:junit4] 2> 124971 T626 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 124972 T626 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 124972 T626 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 124974 T626 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371505389106/collection1/'
[junit4:junit4] 2> 124974 T626 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1371505389106/collection1/lib/README' to classloader
[junit4:junit4] 2> 124974 T626 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1371505389106/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 124996 T626 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 125028 T626 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 125031 T626 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 125035 T626 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 125346 T626 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 125356 T626 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 125358 T626 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 125382 T626 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 125387 T626 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 125391 T626 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 125392 T626 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 125393 T626 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 125393 T626 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 125394 T626 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 125395 T626 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 125395 T626 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 125395 T626 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1371505389106/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty1/
[junit4:junit4] 2> 125396 T626 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@25c22d44
[junit4:junit4] 2> 125396 T626 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 125397 T626 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty1
[junit4:junit4] 2> 125397 T626 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty1/index/
[junit4:junit4] 2> 125397 T626 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 125398 T626 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty1/index
[junit4:junit4] 2> 125399 T626 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c29d68e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6895fc57),segFN=segments_1,generation=1}
[junit4:junit4] 2> 125400 T626 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 125402 T626 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 125402 T626 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 125403 T626 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 125403 T626 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 125403 T626 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 125404 T626 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 125404 T626 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 125404 T626 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 125404 T626 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 125407 T626 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 125408 T626 oass.SolrIndexSearcher.<init> Opening Searcher@22b262e2 main
[junit4:junit4] 2> 125409 T626 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty1/tlog
[junit4:junit4] 2> 125409 T626 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 125409 T626 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 125413 T627 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@22b262e2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 125414 T626 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 125414 T626 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56556/u_ collection:collection1 shard:shard1
[junit4:junit4] 2> 125415 T626 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 125425 T626 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 125429 T626 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 125429 T626 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 125429 T626 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56556/u_/collection1/
[junit4:junit4] 2> 125429 T626 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 125430 T626 oasc.SyncStrategy.syncToMe http://127.0.0.1:56556/u_/collection1/ has no replicas
[junit4:junit4] 2> 125430 T626 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56556/u_/collection1/
[junit4:junit4] 2> 125430 T626 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 125768 T606 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 125777 T605 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 125778 T612 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 125777 T625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 125789 T626 oasc.ZkController.register We are http://127.0.0.1:56556/u_/collection1/ and leader is http://127.0.0.1:56556/u_/collection1/
[junit4:junit4] 2> 125789 T626 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56556/u_
[junit4:junit4] 2> 125789 T626 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 125789 T626 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 125789 T626 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 125792 T626 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 125793 T583 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 125793 T583 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 125794 T583 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 125872 T583 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 125875 T583 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35185
[junit4:junit4] 2> 125875 T583 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 125876 T583 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 125876 T583 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371505392094
[junit4:junit4] 2> 125877 T583 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371505392094/solr.xml
[junit4:junit4] 2> 125877 T583 oasc.CoreContainer.<init> New CoreContainer 1035738333
[junit4:junit4] 2> 125877 T583 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371505392094/'
[junit4:junit4] 2> 125878 T583 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371505392094/'
[junit4:junit4] 2> 125946 T583 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 125947 T583 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 125947 T583 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 125948 T583 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 125948 T583 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 125948 T583 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 125948 T583 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 125949 T583 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 125949 T583 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 125949 T583 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 125953 T583 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 125953 T583 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34322/solr
[junit4:junit4] 2> 125954 T583 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 125954 T583 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 125957 T639 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ebd00b4 name:ZooKeeperConnection Watcher:127.0.0.1:34322 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 125957 T583 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 125959 T583 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 125962 T583 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 125963 T641 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@441b8053 name:ZooKeeperConnection Watcher:127.0.0.1:34322/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 125964 T583 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 125967 T583 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 126971 T583 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35185_u_
[junit4:junit4] 2> 126972 T583 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35185_u_
[junit4:junit4] 2> 126974 T612 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 126975 T605 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 126974 T625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 126975 T641 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 126977 T625 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 126977 T612 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 126978 T605 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 126981 T642 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 126981 T642 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 127284 T606 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 127285 T606 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56556_u_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56556/u_"}
[junit4:junit4] 2> 127290 T606 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35185_u_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35185/u_"}
[junit4:junit4] 2> 127290 T606 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 127290 T606 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 127294 T605 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 127294 T625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 127295 T641 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 127294 T612 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 127982 T642 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371505392094/collection1
[junit4:junit4] 2> 127982 T642 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 127983 T642 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 127983 T642 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 127985 T642 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371505392094/collection1/'
[junit4:junit4] 2> 127986 T642 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1371505392094/collection1/lib/README' to classloader
[junit4:junit4] 2> 127986 T642 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1371505392094/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 128017 T642 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 128050 T642 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 128052 T642 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 128056 T642 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 128377 T642 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 128382 T642 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 128384 T642 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 128399 T642 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 128402 T642 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 128404 T642 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 128405 T642 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 128405 T642 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 128406 T642 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 128407 T642 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 128407 T642 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 128407 T642 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 128408 T642 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1371505392094/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2/
[junit4:junit4] 2> 128409 T642 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@25c22d44
[junit4:junit4] 2> 128409 T642 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 128410 T642 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2
[junit4:junit4] 2> 128410 T642 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2/index/
[junit4:junit4] 2> 128410 T642 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 128410 T642 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2/index
[junit4:junit4] 2> 128412 T642 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@9d01017 lockFactory=org.apache.lucene.store.NativeFSLockFactory@44083847),segFN=segments_1,generation=1}
[junit4:junit4] 2> 128412 T642 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 128413 T642 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 128414 T642 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 128414 T642 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 128414 T642 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 128415 T642 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 128415 T642 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 128415 T642 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 128416 T642 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 128416 T642 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 128418 T642 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 128420 T642 oass.SolrIndexSearcher.<init> Opening Searcher@3245bffe main
[junit4:junit4] 2> 128420 T642 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2/tlog
[junit4:junit4] 2> 128421 T642 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 128421 T642 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 128425 T643 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3245bffe main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 128426 T642 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 128426 T642 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35185/u_ collection:collection1 shard:shard1
[junit4:junit4] 2> 128429 T642 oasc.ZkController.register We are http://127.0.0.1:35185/u_/collection1/ and leader is http://127.0.0.1:56556/u_/collection1/
[junit4:junit4] 2> 128429 T642 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35185/u_
[junit4:junit4] 2> 128429 T642 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 128429 T642 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C982 name=collection1 org.apache.solr.core.SolrCore@7c04c6ec url=http://127.0.0.1:35185/u_/collection1 node=127.0.0.1:35185_u_ C982_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:35185_u_, base_url=http://127.0.0.1:35185/u_}
[junit4:junit4] 2> 128430 T644 C982 P35185 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 128430 T644 C982 P35185 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 128430 T642 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 128430 T644 C982 P35185 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 128431 T644 C982 P35185 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 128431 T583 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 128432 T583 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 128432 T583 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 128432 T644 C982 P35185 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 128441 T583 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 128442 T618 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> ASYNC NEW_CORE C983 name=collection1 org.apache.solr.core.SolrCore@658b0b44 url=http://127.0.0.1:34380/u_/collection1 node=127.0.0.1:34380_u_ C983_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:34380_u_, base_url=http://127.0.0.1:34380/u_, leader=true}
[junit4:junit4] 2> 128451 T599 C983 P34380 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60c8fab lockFactory=org.apache.lucene.store.NativeFSLockFactory@68fbc5d3),segFN=segments_1,generation=1}
[junit4:junit4] 2> 128452 T599 C983 P34380 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 128455 T600 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[1 (1438127640802230273)]} 0 6
[junit4:junit4] 2> 128455 T599 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10001 (1438127640802230272)]} 0 6
[junit4:junit4] 2> ASYNC NEW_CORE C984 name=collection1 org.apache.solr.core.SolrCore@4230efdd url=http://127.0.0.1:56556/u_/collection1 node=127.0.0.1:56556_u_ C984_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:56556_u_, base_url=http://127.0.0.1:56556/u_, leader=true}
[junit4:junit4] 2> 128468 T620 C984 P56556 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c29d68e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6895fc57),segFN=segments_1,generation=1}
[junit4:junit4] 2> 128469 T620 C984 P56556 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 128472 T620 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[1 (1438127640819007488)]} 0 6
[junit4:junit4] 2> 128473 T621 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10001 (1438127640821104640)]} 0 6
[junit4:junit4] 2> 128557 T601 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[2 (1438127640913379328)]} 0 2
[junit4:junit4] 2> 128558 T598 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10002 (1438127640913379329)]} 0 2
[junit4:junit4] 2> 128561 T619 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[2 (1438127640917573632)]} 0 1
[junit4:junit4] 2> 128562 T621 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10002 (1438127640918622208)]} 0 1
[junit4:junit4] 2> 128656 T600 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[3 (1438127641017188352)]} 0 2
[junit4:junit4] 2> 128656 T599 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10003 (1438127641017188353)]} 0 1
[junit4:junit4] 2> 128660 T620 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[3 (1438127641021382656)]} 0 2
[junit4:junit4] 2> 128661 T619 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10003 (1438127641021382657)]} 0 2
[junit4:junit4] 2> 128685 T601 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[4 (1438127641047597056)]} 0 2
[junit4:junit4] 2> 128686 T598 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10004 (1438127641047597057)]} 0 2
[junit4:junit4] 2> 128689 T621 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[4 (1438127641051791360)]} 0 2
[junit4:junit4] 2> 128689 T620 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10004 (1438127641051791361)]} 0 1
[junit4:junit4] 2> 128776 T599 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10005 (1438127641143017473)]} 0 1
[junit4:junit4] 2> 128776 T600 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[5 (1438127641143017472)]} 0 1
[junit4:junit4] 2> 128780 T621 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[5 (1438127641147211776)]} 0 1
[junit4:junit4] 2> 128781 T619 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10005 (1438127641147211777)]} 0 2
[junit4:junit4] 2> 128803 T606 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 128804 T606 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35185_u_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35185/u_"}
[junit4:junit4] 2> 128809 T605 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 128809 T625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 128809 T641 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 128809 T612 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 128854 T601 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[6 (1438127641224806400)]} 0 2
[junit4:junit4] 2> 128854 T599 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10006 (1438127641224806401)]} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C985 name=collection1 org.apache.solr.core.SolrCore@7c04c6ec url=http://127.0.0.1:35185/u_/collection1 node=127.0.0.1:35185_u_ C985_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:35185_u_, base_url=http://127.0.0.1:35185/u_}
[junit4:junit4] 2> 128869 T635 C985 P35185 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@9d01017 lockFactory=org.apache.lucene.store.NativeFSLockFactory@44083847),segFN=segments_1,generation=1}
[junit4:junit4] 2> 128870 T635 C985 P35185 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 128872 T635 C985 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10006 (1438127641229000704)]} 0 4
[junit4:junit4] 2> 128873 T621 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10006 (1438127641229000704)]} 0 16
[junit4:junit4] 2> 128873 T634 C985 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[6 (1438127641227952128)]} 0 3
[junit4:junit4] 2> 128874 T620 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[6 (1438127641227952128)]} 0 18
[junit4:junit4] 2> 128946 T598 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10007 (1438127641320226816)]} 0 2
[junit4:junit4] 2> 128946 T600 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[7 (1438127641321275392)]} 0 1
[junit4:junit4] 2> 128955 T636 C985 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10007 (1438127641325469696)]} 0 3
[junit4:junit4] 2> 128956 T619 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10007 (1438127641325469696)]} 0 8
[junit4:junit4] 2> 128957 T637 C985 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[7 (1438127641325469697)]} 0 4
[junit4:junit4] 2> 128958 T621 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[7 (1438127641325469697)]} 0 9
[junit4:junit4] 2> 129037 T601 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10006 (-1438127641417744384)]} 0 0
[junit4:junit4] 2> 129038 T599 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[6 (-1438127641418792960)]} 0 0
[junit4:junit4] 2> 129043 T635 C985 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10006 (-1438127641420890112)]} 0 0
[junit4:junit4] 2> 129044 T634 C985 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[6 (-1438127641421938688)]} 0 0
[junit4:junit4] 2> 129044 T620 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10006 (-1438127641420890112)]} 0 4
[junit4:junit4] 2> 129045 T621 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[6 (-1438127641421938688)]} 0 4
[junit4:junit4] 2> 129049 T600 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[8 (1438127641428230144)]} 0 2
[junit4:junit4] 2> 129049 T598 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10008 (1438127641428230145)]} 0 2
[junit4:junit4] 2> 129057 T636 C985 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10008 (1438127641433473024)]} 0 2
[junit4:junit4] 2> 129057 T619 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10008 (1438127641433473024)]} 0 5
[junit4:junit4] 2> 129060 T637 C985 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[8 (1438127641433473025)]} 0 4
[junit4:junit4] 2> 129061 T620 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[8 (1438127641433473025)]} 0 9
[junit4:junit4] 2> 129130 T601 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10007 (-1438127641515261952)]} 0 0
[junit4:junit4] 2> 129132 T599 C983 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[7 (-1438127641517359104)]} 0 0
[junit4:junit4] 2> 129136 T635 C985 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10007 (-1438127641518407680)]} 0 0
[junit4:junit4] 2> 129137 T621 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10007 (-1438127641518407680)]} 0 4
[junit4:junit4] 2> 129137 T636 C985 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[7 (-1438127641520504832)]} 0 0
[junit4:junit4] 2> 129138 T619 C984 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[7 (-1438127641520504832)]} 0 3
[junit4:junit4] 2> ASYNC NEW_CORE C986 name=collection1 org.apache.solr.core.SolrCore@658b0b44 url=http://127.0.0.1:34380/u_/collection1 node=127.0.0.1:34380_u_ C986_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:34380_u_, base_url=http://127.0.0.1:34380/u_, leader=true}
[junit4:junit4] 2> 129150 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10009 (1438127641524699136)]} 0 11
[junit4:junit4] 2> 129151 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[9 (1438127641536233472)]} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C987 name=collection1 org.apache.solr.core.SolrCore@7c04c6ec url=http://127.0.0.1:35185/u_/collection1 node=127.0.0.1:35185_u_ C987_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:35185_u_, base_url=http://127.0.0.1:35185/u_}
[junit4:junit4] 2> 129158 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10009 (1438127641539379200)]} 0 2
[junit4:junit4] 2> ASYNC NEW_CORE C988 name=collection1 org.apache.solr.core.SolrCore@4230efdd url=http://127.0.0.1:56556/u_/collection1 node=127.0.0.1:56556_u_ C988_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:56556_u_, base_url=http://127.0.0.1:56556/u_, leader=true}
[junit4:junit4] 2> 129158 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10009 (1438127641539379200)]} 0 5
[junit4:junit4] 2> 129159 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[9 (1438127641540427776)]} 0 2
[junit4:junit4] 2> 129159 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[9 (1438127641540427776)]} 0 5
[junit4:junit4] 2> 129260 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10 (1438127641650528257)]} 0 1
[junit4:junit4] 2> 129260 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10010 (1438127641650528256)]} 0 2
[junit4:junit4] 2> 129267 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10 (1438127641654722560)]} 0 1
[junit4:junit4] 2> 129268 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10 (1438127641654722560)]} 0 5
[junit4:junit4] 2> 129268 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10010 (1438127641654722561)]} 0 1
[junit4:junit4] 2> 129269 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10010 (1438127641654722561)]} 0 6
[junit4:junit4] 2> 129348 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[11 (1438127641741754368)]} 0 2
[junit4:junit4] 2> 129349 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10011 (1438127641742802944)]} 0 3
[junit4:junit4] 2> 129355 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[11 (1438127641746997248)]} 0 2
[junit4:junit4] 2> 129356 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[11 (1438127641746997248)]} 0 6
[junit4:junit4] 2> 129356 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10011 (1438127641746997249)]} 0 2
[junit4:junit4] 2> 129357 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10011 (1438127641746997249)]} 0 6
[junit4:junit4] 2> 129427 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[8 (-1438127641826689024)]} 0 0
[junit4:junit4] 2> 129427 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10008 (-1438127641826689025)]} 0 0
[junit4:junit4] 2> 129432 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[8 (-1438127641829834752)]} 0 0
[junit4:junit4] 2> 129432 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10008 (-1438127641829834753)]} 0 0
[junit4:junit4] 2> 129433 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[8 (-1438127641829834752)]} 0 4
[junit4:junit4] 2> 129433 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10008 (-1438127641829834753)]} 0 3
[junit4:junit4] 2> 129437 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[12 (1438127641836126208)]} 0 2
[junit4:junit4] 2> 129438 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10012 (1438127641836126209)]} 0 3
[junit4:junit4] 2> 129443 T618 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 129443 T618 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=3&state=recovering&nodeName=127.0.0.1:35185_u_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 129444 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1438127641840320512)]} 0 1
[junit4:junit4] 2> 129445 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[12 (1438127641840320512)]} 0 6
[junit4:junit4] 2> 129445 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10012 (1438127641840320513)]} 0 2
[junit4:junit4] 2> 129446 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10012 (1438127641840320513)]} 0 6
[junit4:junit4] 2> 129503 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10009 (-1438127641906380801)]} 0 0
[junit4:junit4] 2> 129503 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[9 (-1438127641906380800)]} 0 1
[junit4:junit4] 2> 129508 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10009 (-1438127641909526528)]} 0 0
[junit4:junit4] 2> 129509 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[9 (-1438127641909526529)]} 0 1
[junit4:junit4] 2> 129509 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10009 (-1438127641909526528)]} 0 3
[junit4:junit4] 2> 129509 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[9 (-1438127641909526529)]} 0 3
[junit4:junit4] 2> 129513 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10013 (1438127641915817984)]} 0 2
[junit4:junit4] 2> 129513 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[13 (1438127641915817985)]} 0 2
[junit4:junit4] 2> 129520 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10013 (1438127641918963712)]} 0 1
[junit4:junit4] 2> 129521 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1438127641920012288)]} 0 2
[junit4:junit4] 2> 129521 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10013 (1438127641918963712)]} 0 6
[junit4:junit4] 2> 129522 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[13 (1438127641920012288)]} 0 6
[junit4:junit4] 2> 129567 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[11 (-1438127641972441088)]} 0 1
[junit4:junit4] 2> 129567 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10011 (-1438127641972441089)]} 0 1
[junit4:junit4] 2> 129572 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10011 (-1438127641976635393)]} 0 0
[junit4:junit4] 2> 129572 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[11 (-1438127641976635392)]} 0 0
[junit4:junit4] 2> 129573 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10011 (-1438127641976635393)]} 0 4
[junit4:junit4] 2> 129573 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[11 (-1438127641976635392)]} 0 4
[junit4:junit4] 2> 129577 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10014 (1438127641982926848)]} 0 2
[junit4:junit4] 2> 129577 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[14 (1438127641982926849)]} 0 2
[junit4:junit4] 2> 129584 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10014 (1438127641986072576)]} 0 1
[junit4:junit4] 2> 129585 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1438127641987121152)]} 0 2
[junit4:junit4] 2> 129585 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10014 (1438127641986072576)]} 0 6
[junit4:junit4] 2> 129585 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[14 (1438127641987121152)]} 0 5
[junit4:junit4] 2> 129592 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10015 (1438127641998655488)]} 0 1
[junit4:junit4] 2> 129592 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[15 (1438127641998655489)]} 0 1
[junit4:junit4] 2> 129599 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10015 (1438127642002849792)]} 0 1
[junit4:junit4] 2> 129600 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1438127642002849793)]} 0 1
[junit4:junit4] 2> 129600 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10015 (1438127642002849792)]} 0 5
[junit4:junit4] 2> 129600 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[15 (1438127642002849793)]} 0 5
[junit4:junit4] 2> 129668 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10012 (-1438127642079395840)]} 0 1
[junit4:junit4] 2> 129668 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[12 (-1438127642079395841)]} 0 0
[junit4:junit4] 2> 129676 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10012 (-1438127642083590144)]} 0 1
[junit4:junit4] 2> 129676 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[12 (-1438127642084638720)]} 0 0
[junit4:junit4] 2> 129677 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10012 (-1438127642083590144)]} 0 5
[junit4:junit4] 2> 129678 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[12 (-1438127642084638720)]} 0 6
[junit4:junit4] 2> 129684 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[16 (1438127642094075905)]} 0 3
[junit4:junit4] 2> 129684 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10016 (1438127642094075904)]} 0 3
[junit4:junit4] 2> 129690 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1438127642098270208)]} 0 1
[junit4:junit4] 2> 129691 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10016 (1438127642099318784)]} 0 1
[junit4:junit4] 2> 129691 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[16 (1438127642098270208)]} 0 5
[junit4:junit4] 2> 129691 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10016 (1438127642099318784)]} 0 5
[junit4:junit4] 2> 129714 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[17 (1438127642126581761)]} 0 1
[junit4:junit4] 2> 129715 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10017 (1438127642126581760)]} 0 2
[junit4:junit4] 2> 129721 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1438127642130776064)]} 0 1
[junit4:junit4] 2> 129722 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10017 (1438127642130776065)]} 0 2
[junit4:junit4] 2> 129722 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[17 (1438127642130776064)]} 0 5
[junit4:junit4] 2> 129722 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10017 (1438127642130776065)]} 0 5
[junit4:junit4] 2> 129769 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[14 (-1438127642185302016)]} 0 0
[junit4:junit4] 2> 129769 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10014 (-1438127642185302017)]} 0 0
[junit4:junit4] 2> 129774 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[14 (-1438127642188447744)]} 0 0
[junit4:junit4] 2> 129774 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10014 (-1438127642188447745)]} 0 0
[junit4:junit4] 2> 129775 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[14 (-1438127642188447744)]} 0 3
[junit4:junit4] 2> 129775 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10014 (-1438127642188447745)]} 0 3
[junit4:junit4] 2> 129778 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[18 (1438127642193690624)]} 0 1
[junit4:junit4] 2> 129779 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10018 (1438127642193690625)]} 0 2
[junit4:junit4] 2> 129785 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1438127642197884928)]} 0 1
[junit4:junit4] 2> 129786 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[18 (1438127642197884928)]} 0 6
[junit4:junit4] 2> 129786 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10018 (1438127642197884929)]} 0 2
[junit4:junit4] 2> 129787 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10018 (1438127642197884929)]} 0 6
[junit4:junit4] 2> 129840 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10019 (1438127642257653761)]} 0 2
[junit4:junit4] 2> 129840 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[19 (1438127642257653760)]} 0 3
[junit4:junit4] 2> 129849 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10019 (1438127642263945216)]} 0 2
[junit4:junit4] 2> 129850 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1438127642263945217)]} 0 2
[junit4:junit4] 2> 129850 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10019 (1438127642263945216)]} 0 6
[junit4:junit4] 2> 129850 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[19 (1438127642263945217)]} 0 6
[junit4:junit4] 2> 129934 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10015 (-1438127642358317056)]} 0 0
[junit4:junit4] 2> 129934 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[15 (-1438127642358317057)]} 0 0
[junit4:junit4] 2> 129939 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10015 (-1438127642361462784)]} 0 0
[junit4:junit4] 2> 129940 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[15 (-1438127642361462785)]} 0 1
[junit4:junit4] 2> 129940 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10015 (-1438127642361462784)]} 0 3
[junit4:junit4] 2> 129940 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[15 (-1438127642361462785)]} 0 3
[junit4:junit4] 2> 129943 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10020 (1438127642366705664)]} 0 1
[junit4:junit4] 2> 129943 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[20 (1438127642367754240)]} 0 1
[junit4:junit4] 2> 129951 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1438127642370899969)]} 0 2
[junit4:junit4] 2> 129951 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10020 (1438127642370899968)]} 0 1
[junit4:junit4] 2> 129951 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[20 (1438127642370899969)]} 0 5
[junit4:junit4] 2> 129952 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10020 (1438127642370899968)]} 0 6
[junit4:junit4] 2> 130010 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10021 (1438127642438008832)]} 0 1
[junit4:junit4] 2> 130011 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[21 (1438127642438008833)]} 0 2
[junit4:junit4] 2> 130017 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10021 (1438127642441154560)]} 0 1
[junit4:junit4] 2> 130018 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1438127642441154561)]} 0 2
[junit4:junit4] 2> 130018 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10021 (1438127642441154560)]} 0 5
[junit4:junit4] 2> 130018 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[21 (1438127642441154561)]} 0 5
[junit4:junit4] 2> 130060 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10022 (1438127642489389057)]} 0 1
[junit4:junit4] 2> 130060 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[22 (1438127642489389056)]} 0 1
[junit4:junit4] 2> 130067 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10022 (1438127642493583360)]} 0 1
[junit4:junit4] 2> 130068 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1438127642493583361)]} 0 2
[junit4:junit4] 2> 130068 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10022 (1438127642493583360)]} 0 5
[junit4:junit4] 2> 130068 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[22 (1438127642493583361)]} 0 5
[junit4:junit4] 2> 130080 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[23 (1438127642510360577)]} 0 1
[junit4:junit4] 2> 130080 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10023 (1438127642510360576)]} 0 1
[junit4:junit4] 2> 130087 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1438127642514554880)]} 0 1
[junit4:junit4] 2> 130088 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10023 (1438127642514554881)]} 0 2
[junit4:junit4] 2> 130088 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[23 (1438127642514554880)]} 0 5
[junit4:junit4] 2> 130089 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10023 (1438127642514554881)]} 0 6
[junit4:junit4] 2> 130156 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10016 (-1438127642590052352)]} 0 1
[junit4:junit4] 2> 130156 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[16 (-1438127642590052353)]} 0 1
[junit4:junit4] 2> 130161 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10016 (-1438127642594246656)]} 0 0
[junit4:junit4] 2> 130161 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[16 (-1438127642594246657)]} 0 0
[junit4:junit4] 2> 130162 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10016 (-1438127642594246656)]} 0 4
[junit4:junit4] 2> 130162 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[16 (-1438127642594246657)]} 0 4
[junit4:junit4] 2> 130166 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[24 (1438127642600538113)]} 0 1
[junit4:junit4] 2> 130166 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10024 (1438127642600538112)]} 0 1
[junit4:junit4] 2> 130172 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1438127642604732416)]} 0 1
[junit4:junit4] 2> 130173 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10024 (1438127642604732417)]} 0 1
[junit4:junit4] 2> 130173 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[24 (1438127642604732416)]} 0 5
[junit4:junit4] 2> 130173 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10024 (1438127642604732417)]} 0 4
[junit4:junit4] 2> 130210 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[25 (1438127642646675456)]} 0 1
[junit4:junit4] 2> 130211 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10025 (1438127642646675457)]} 0 2
[junit4:junit4] 2> 130217 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1438127642650869760)]} 0 1
[junit4:junit4] 2> 130218 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10025 (1438127642650869761)]} 0 2
[junit4:junit4] 2> 130218 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[25 (1438127642650869760)]} 0 5
[junit4:junit4] 2> 130219 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10025 (1438127642650869761)]} 0 6
[junit4:junit4] 2> 130252 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[17 (-1438127642691764224)]} 0 0
[junit4:junit4] 2> 130252 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10017 (-1438127642691764225)]} 0 0
[junit4:junit4] 2> 130257 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[17 (-1438127642694909952)]} 0 0
[junit4:junit4] 2> 130257 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10017 (-1438127642694909953)]} 0 0
[junit4:junit4] 2> 130258 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[17 (-1438127642694909952)]} 0 3
[junit4:junit4] 2> 130258 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10017 (-1438127642694909953)]} 0 3
[junit4:junit4] 2> 130262 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10026 (1438127642701201409)]} 0 2
[junit4:junit4] 2> 130262 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[26 (1438127642701201408)]} 0 2
[junit4:junit4] 2> 130269 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10026 (1438127642704347136)]} 0 2
[junit4:junit4] 2> 130269 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1438127642705395712)]} 0 1
[junit4:junit4] 2> 130269 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10026 (1438127642704347136)]} 0 5
[junit4:junit4] 2> 130270 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[26 (1438127642705395712)]} 0 5
[junit4:junit4] 2> 130361 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[18 (-1438127642806059008)]} 0 1
[junit4:junit4] 2> 130361 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10018 (-1438127642806059009)]} 0 1
[junit4:junit4] 2> 130366 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[18 (-1438127642809204736)]} 0 0
[junit4:junit4] 2> 130366 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10018 (-1438127642809204737)]} 0 0
[junit4:junit4] 2> 130366 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[18 (-1438127642809204736)]} 0 3
[junit4:junit4] 2> 130367 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10018 (-1438127642809204737)]} 0 3
[junit4:junit4] 2> 130370 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[27 (1438127642814447616)]} 0 1
[junit4:junit4] 2> 130371 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10027 (1438127642815496192)]} 0 2
[junit4:junit4] 2> 130377 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1438127642818641920)]} 0 1
[junit4:junit4] 2> 130378 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10027 (1438127642818641921)]} 0 2
[junit4:junit4] 2> 130378 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[27 (1438127642818641920)]} 0 5
[junit4:junit4] 2> 130378 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10027 (1438127642818641921)]} 0 5
[junit4:junit4] 2> 130417 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10019 (-1438127642864779264)]} 0 0
[junit4:junit4] 2> 130417 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[19 (-1438127642864779265)]} 0 0
[junit4:junit4] 2> 130424 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10019 (-1438127642867924992)]} 0 1
[junit4:junit4] 2> 130424 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[19 (-1438127642867924993)]} 0 1
[junit4:junit4] 2> 130424 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10019 (-1438127642867924992)]} 0 5
[junit4:junit4] 2> 130425 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[19 (-1438127642867924993)]} 0 5
[junit4:junit4] 2> 130429 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10028 (1438127642876313601)]} 0 2
[junit4:junit4] 2> 130429 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[28 (1438127642876313600)]} 0 2
[junit4:junit4] 2> 130436 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1438127642880507905)]} 0 1
[junit4:junit4] 2> 130437 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10028 (1438127642880507904)]} 0 1
[junit4:junit4] 2> 130437 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[28 (1438127642880507905)]} 0 5
[junit4:junit4] 2> 130438 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10028 (1438127642880507904)]} 0 6
[junit4:junit4] 2> 130444 T583 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 35185
[junit4:junit4] 2> 130444 T583 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1035738333
[junit4:junit4] 2> 130510 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[29 (1438127642962296832)]} 0 1
[junit4:junit4] 2> 130510 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10029 (1438127642962296833)]} 0 1
[junit4:junit4] 2> 130518 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1438127642966491137)]} 0 1
[junit4:junit4] 2> 130519 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10029 (1438127642966491136)]} 0 2
[junit4:junit4] 2> 130519 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[29 (1438127642966491137)]} 0 6
[junit4:junit4] 2> 130519 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10029 (1438127642966491136)]} 0 6
[junit4:junit4] 2> 130575 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10030 (1438127643029405696)]} 0 1
[junit4:junit4] 2> 130576 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[30 (1438127643029405697)]} 0 2
[junit4:junit4] 2> 130583 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1438127643034648576)]} 0 1
[junit4:junit4] 2> 130584 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10030 (1438127643033600000)]} 0 2
[junit4:junit4] 2> 130584 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[30 (1438127643034648576)]} 0 6
[junit4:junit4] 2> 130584 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10030 (1438127643033600000)]} 0 6
[junit4:junit4] 2> 130683 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10031 (1438127643142651904)]} 0 2
[junit4:junit4] 2> 130683 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[31 (1438127643142651905)]} 0 2
[junit4:junit4] 2> 130691 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10031 (1438127643146846208)]} 0 2
[junit4:junit4] 2> 130691 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1438127643146846209)]} 0 1
[junit4:junit4] 2> 130692 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10031 (1438127643146846208)]} 0 6
[junit4:junit4] 2> 130692 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[31 (1438127643146846209)]} 0 6
[junit4:junit4] 2> 130762 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10032 (1438127643225489408)]} 0 1
[junit4:junit4] 2> 130763 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[32 (1438127643225489409)]} 0 1
[junit4:junit4] 2> 130769 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10032 (1438127643229683712)]} 0 1
[junit4:junit4] 2> 130769 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1438127643229683713)]} 0 1
[junit4:junit4] 2> 130770 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10032 (1438127643229683712)]} 0 5
[junit4:junit4] 2> 130770 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[32 (1438127643229683713)]} 0 5
[junit4:junit4] 2> 130859 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10033 (1438127643327201281)]} 0 1
[junit4:junit4] 2> 130859 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[33 (1438127643327201280)]} 0 1
[junit4:junit4] 2> 130867 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1438127643332444160)]} 0 1
[junit4:junit4] 2> 130868 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10033 (1438127643332444161)]} 0 2
[junit4:junit4] 2> 130868 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[33 (1438127643332444160)]} 0 6
[junit4:junit4] 2> 130868 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10033 (1438127643332444161)]} 0 6
[junit4:junit4] 2> 130949 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[34 (1438127643420524544)]} 0 3
[junit4:junit4] 2> 130950 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10034 (1438127643420524545)]} 0 3
[junit4:junit4] 2> 130959 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1438127643426816000)]} 0 2
[junit4:junit4] 2> 130961 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10034 (1438127643427864576)]} 0 3
[junit4:junit4] 2> 130961 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[34 (1438127643426816000)]} 0 8
[junit4:junit4] 2> 130962 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10034 (1438127643427864576)]} 0 9
[junit4:junit4] 2> 131001 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[22 (-1438127643477147648)]} 0 0
[junit4:junit4] 2> 131002 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10022 (-1438127643478196224)]} 0 0
[junit4:junit4] 2> 131006 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[22 (-1438127643480293376)]} 0 0
[junit4:junit4] 2> 131006 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10022 (-1438127643480293377)]} 0 0
[junit4:junit4] 2> 131007 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[22 (-1438127643480293376)]} 0 3
[junit4:junit4] 2> 131007 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10022 (-1438127643480293377)]} 0 3
[junit4:junit4] 2> 131012 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[35 (1438127643486584832)]} 0 3
[junit4:junit4] 2> 131012 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10035 (1438127643486584833)]} 0 2
[junit4:junit4] 2> 131019 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1438127643491827712)]} 0 1
[junit4:junit4] 2> 131020 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10035 (1438127643491827713)]} 0 2
[junit4:junit4] 2> 131020 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[35 (1438127643491827712)]} 0 6
[junit4:junit4] 2> 131021 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10035 (1438127643491827713)]} 0 6
[junit4:junit4] 2> 131069 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10036 (1438127643547402241)]} 0 1
[junit4:junit4] 2> 131069 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[36 (1438127643547402240)]} 0 1
[junit4:junit4] 2> 131077 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1438127643552645120)]} 0 1
[junit4:junit4] 2> 131078 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10036 (1438127643552645121)]} 0 2
[junit4:junit4] 2> 131078 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[36 (1438127643552645120)]} 0 6
[junit4:junit4] 2> 131079 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10036 (1438127643552645121)]} 0 7
[junit4:junit4] 2> 131146 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10037 (1438127643628142592)]} 0 2
[junit4:junit4] 2> 131146 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[37 (1438127643628142593)]} 0 2
[junit4:junit4] 2> 131154 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10037 (1438127643633385472)]} 0 1
[junit4:junit4] 2> 131155 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1438127643633385473)]} 0 2
[junit4:junit4] 2> 131155 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10037 (1438127643633385472)]} 0 5
[junit4:junit4] 2> 131155 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[37 (1438127643633385473)]} 0 5
[junit4:junit4] 2> 131163 T601 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[24 (-1438127643647016960)]} 0 0
[junit4:junit4] 2> 131163 T600 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10024 (-1438127643647016961)]} 0 0
[junit4:junit4] 2> 131169 T634 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[24 (-1438127643650162688)]} 0 1
[junit4:junit4] 2> 131170 T618 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[24 (-1438127643650162688)]} 0 4
[junit4:junit4] 2> 131170 T635 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10024 (-1438127643651211264)]} 0 1
[junit4:junit4] 2> 131171 T619 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10024 (-1438127643651211264)]} 0 4
[junit4:junit4] 2> 131174 T599 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[38 (1438127643656454144)]} 0 2
[junit4:junit4] 2> 131174 T598 C986 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10038 (1438127643657502720)]} 0 1
[junit4:junit4] 2> 131181 T637 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10038 (1438127643661697024)]} 0 1
[junit4:junit4] 2> 131182 T636 C987 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[38 (1438127643661697025)]} 0 2
[junit4:junit4] 2> 131183 T620 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[38 (1438127643661697025)]} 0 6
[junit4:junit4] 2> 131183 T621 C988 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10038 (1438127643661697024)]} 0 6
[junit4:junit4] 2> ASYNC NEW_CORE C989 name=collection1 org.apache.solr.core.SolrCore@658b0b44 url=http://127.0.0.1:34380/u_/collection1 node=127.0.0.1:34380_u_ C989_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:34380_u_, base_url=http://127.0.0.1:34380/u_, leader=true}
[junit4:junit4] 2> 131251 T601 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[39 (1438127643738243072)]} 0 1
[junit4:junit4] 2> 131252 T600 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10039 (1438127643739291648)]} 0 2
[junit4:junit4] 2> ASYNC NEW_CORE C990 name=collection1 org.apache.solr.core.SolrCore@7c04c6ec url=http://127.0.0.1:35185/u_/collection1 node=127.0.0.1:35185_u_ C990_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:35185_u_, base_url=http://127.0.0.1:35185/u_}
[junit4:junit4] 2> 131259 T634 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[39 (1438127643743485953)]} 0 1
[junit4:junit4] 2> 131260 T635 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10039 (1438127643743485952)]} 0 2
[junit4:junit4] 2> ASYNC NEW_CORE C991 name=collection1 org.apache.solr.core.SolrCore@4230efdd url=http://127.0.0.1:56556/u_/collection1 node=127.0.0.1:56556_u_ C991_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:56556_u_, base_url=http://127.0.0.1:56556/u_, leader=true}
[junit4:junit4] 2> 131260 T618 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[39 (1438127643743485953)]} 0 6
[junit4:junit4] 2> 131261 T619 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10039 (1438127643743485952)]} 0 7
[junit4:junit4] 2> 131336 T598 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[25 (-1438127643828420608)]} 0 0
[junit4:junit4] 2> 131337 T599 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10025 (-1438127643828420609)]} 0 1
[junit4:junit4] 2> 131342 T636 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[25 (-1438127643831566336)]} 0 1
[junit4:junit4] 2> 131342 T637 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10025 (-1438127643831566337)]} 0 1
[junit4:junit4] 2> 131343 T621 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[25 (-1438127643831566336)]} 0 3
[junit4:junit4] 2> 131343 T620 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10025 (-1438127643831566337)]} 0 4
[junit4:junit4] 2> 131346 T601 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[40 (1438127643837857792)]} 0 1
[junit4:junit4] 2> 131346 T600 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10040 (1438127643837857793)]} 0 1
[junit4:junit4] 2> 131354 T634 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[40 (1438127643842052096)]} 0 2
[junit4:junit4] 2> 131354 T635 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10040 (1438127643843100672)]} 0 1
[junit4:junit4] 2> 131355 T618 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[40 (1438127643842052096)]} 0 6
[junit4:junit4] 2> 131355 T619 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10040 (1438127643843100672)]} 0 6
[junit4:junit4] 2> 131401 T601 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[26 (-1438127643896578048)]} 0 1
[junit4:junit4] 2> 131401 T598 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10026 (-1438127643896578049)]} 0 0
[junit4:junit4] 2> 131406 T636 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10026 (-1438127643899723776)]} 0 0
[junit4:junit4] 2> 131407 T637 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[26 (-1438127643899723777)]} 0 1
[junit4:junit4] 2> 131407 T620 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10026 (-1438127643899723776)]} 0 3
[junit4:junit4] 2> 131408 T621 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[26 (-1438127643899723777)]} 0 4
[junit4:junit4] 2> 131411 T600 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10041 (1438127643906015232)]} 0 1
[junit4:junit4] 2> 131412 T599 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[41 (1438127643907063808)]} 0 2
[junit4:junit4] 2> 131419 T635 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[41 (1438127643910209537)]} 0 1
[junit4:junit4] 2> 131419 T619 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[41 (1438127643910209537)]} 0 5
[junit4:junit4] 2> 131420 T634 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10041 (1438127643910209536)]} 0 3
[junit4:junit4] 2> 131421 T618 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10041 (1438127643910209536)]} 0 7
[junit4:junit4] 2> 131443 T644 C990 P35185 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:56556/u_/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 131444 T644 C990 P35185 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:35185/u_ START replicas=[http://127.0.0.1:56556/u_/collection1/] nUpdates=100
[junit4:junit4] 2> 131449 T620 C991 P56556 oasc.SolrCore.execute [collection1] webapp=/u_ path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=3
[junit4:junit4] 2> 131449 T644 C990 P35185 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 131450 T644 C990 P35185 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 131450 T644 C990 P35185 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 131450 T644 C990 P35185 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 131450 T644 C990 P35185 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2/tlog/tlog.0000000000000000000 refcount=1}}
[junit4:junit4] 2> 131451 T644 C990 P35185 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:56556/u_/collection1/. core=collection1
[junit4:junit4] 2> 131451 T644 C990 P35185 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 131455 T621 C991 P56556 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 131475 T621 C991 P56556 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c29d68e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6895fc57),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c29d68e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6895fc57),segFN=segments_2,generation=2}
[junit4:junit4] 2> 131476 T621 C991 P56556 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 131481 T621 C991 P56556 oass.SolrIndexSearcher.<init> Opening Searcher@4959d0ad realtime
[junit4:junit4] 2> 131482 T621 C991 P56556 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 131482 T621 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 27
[junit4:junit4] 2> 131483 T644 C990 P35185 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 131483 T644 C990 P35185 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 131486 T618 C991 P56556 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 131486 T618 C991 P56556 oasc.SolrCore.execute [collection1] webapp=/u_ path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 131487 T644 C990 P35185 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 131487 T644 C990 P35185 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 131487 T644 C990 P35185 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 131489 T620 C991 P56556 oasc.SolrCore.execute [collection1] webapp=/u_ path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 131490 T644 C990 P35185 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 5
[junit4:junit4] 2> 131491 T644 C990 P35185 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2/index.20130617124317788
[junit4:junit4] 2> 131491 T644 C990 P35185 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@55a9f594 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1b96ff4e) fullCopy=false
[junit4:junit4] 2> 131494 T621 C991 P56556 oasc.SolrCore.execute [collection1] webapp=/u_ path=/replication params={file=_0.cfs&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 131497 T619 C991 P56556 oasc.SolrCore.execute [collection1] webapp=/u_ path=/replication params={file=_0.cfe&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 131498 T601 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10042 (1438127643997241344)]} 0 1
[junit4:junit4] 2> 131501 T598 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[42 (1438127643999338496)]} 0 2
[junit4:junit4] 2> 131501 T618 C991 P56556 oasc.SolrCore.execute [collection1] webapp=/u_ path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 131505 T619 C991 P56556 oasc.SolrCore.execute [collection1] webapp=/u_ path=/replication params={file=_0_1.del&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 131507 T636 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10042 (1438127644002484224)]} 0 1
[junit4:junit4] 2> 131508 T620 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10042 (1438127644002484224)]} 0 7
[junit4:junit4] 2> 131508 T618 C991 P56556 oasc.SolrCore.execute [collection1] webapp=/u_ path=/replication params={file=_0.si&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 131509 T637 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[42 (1438127644004581376)]} 0 1
[junit4:junit4] 2> 131509 T644 C990 P35185 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 131509 T621 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[42 (1438127644004581376)]} 0 5
[junit4:junit4] 2> 131510 T644 C990 P35185 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 131510 T644 C990 P35185 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 131511 T644 C990 P35185 oasu.DefaultSolrCoreState.newIndexWriter Closing old IndexWriter... core=collection1
[junit4:junit4] 2> 131529 T644 C990 P35185 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@9d01017 lockFactory=org.apache.lucene.store.NativeFSLockFactory@44083847),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@9d01017 lockFactory=org.apache.lucene.store.NativeFSLockFactory@44083847),segFN=segments_2,generation=2}
[junit4:junit4] 2> 131529 T644 C990 P35185 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 131532 T644 C990 P35185 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@9d01017 lockFactory=org.apache.lucene.store.NativeFSLockFactory@44083847),segFN=segments_2,generation=2}
[junit4:junit4] 2> 131532 T644 C990 P35185 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 131532 T644 C990 P35185 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 131534 T644 C990 P35185 oass.SolrIndexSearcher.<init> Opening Searcher@2d4e985a main
[junit4:junit4] 2> 131535 T643 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2d4e985a main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c72/32)}
[junit4:junit4] 2> 131535 T644 C990 P35185 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2/index.20130617124317788 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2/index.20130617124317788;done=true>>]
[junit4:junit4] 2> 131536 T644 C990 P35185 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2/index.20130617124317788
[junit4:junit4] 2> 131536 T644 C990 P35185 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2/index.20130617124317788
[junit4:junit4] 2> 131537 T600 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10043 (1438127644038135808)]} 0 1
[junit4:junit4] 2> 131538 T599 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[43 (1438127644039184384)]} 0 1
[junit4:junit4] 2> 131538 T644 C990 P35185 oasc.RecoveryStrategy.replay Replaying buffered documents. core=collection1
[junit4:junit4] 2> 131539 T652 C990 P35185 oasu.UpdateLog$LogReplayer.doReplay WARN Starting log replay tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2/tlog/tlog.0000000000000000000 refcount=2} active=true starting pos=8105
[junit4:junit4] 2> 131543 T652 C990 P35185 oasu.DirectUpdateHandler2.commit start commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 131548 T652 C990 P35185 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@9d01017 lockFactory=org.apache.lucene.store.NativeFSLockFactory@44083847),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@9d01017 lockFactory=org.apache.lucene.store.NativeFSLockFactory@44083847),segFN=segments_3,generation=3}
[junit4:junit4] 2> 131548 T652 C990 P35185 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 131550 T652 C990 P35185 oass.SolrIndexSearcher.<init> Opening Searcher@491e693a main
[junit4:junit4] 2> 131551 T652 C990 P35185 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 131551 T643 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@491e693a main{StandardDirectoryReader(segments_3:5:nrt _0(5.0):c72/32 _1(5.0):c2)}
[junit4:junit4] 2> 131552 T652 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] {add=[10042 (1438127644002484224), 42 (1438127644004581376)]} 0 14
[junit4:junit4] 2> 131552 T652 C990 P35185 oasu.UpdateLog$LogReplayer.run WARN Log replay finished. recoveryInfo=RecoveryInfo{adds=2 deletes=0 deleteByQuery=0 errors=0 positionOfStart=8105}
[junit4:junit4] 2> 131553 T644 C990 P35185 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 131553 T644 C990 P35185 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 131553 T644 C990 P35185 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 131554 T634 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[43 (1438127644043378689)]} 0 9
[junit4:junit4] 2> 131554 T644 C990 P35185 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 131554 T635 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10043 (1438127644043378688)]} 0 10
[junit4:junit4] 2> 131555 T620 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[43 (1438127644043378689)]} 0 14
[junit4:junit4] 2> 131556 T619 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10043 (1438127644043378688)]} 0 15
[junit4:junit4] 2> 131573 T598 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[44 (1438127644075884544)]} 0 2
[junit4:junit4] 2> 131574 T601 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10044 (1438127644075884545)]} 0 3
[junit4:junit4] 2> 131581 T636 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[44 (1438127644080078848)]} 0 1
[junit4:junit4] 2> 131581 T637 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10044 (1438127644081127424)]} 0 1
[junit4:junit4] 2> 131582 T620 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[44 (1438127644080078848)]} 0 6
[junit4:junit4] 2> 131582 T618 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10044 (1438127644081127424)]} 0 6
[junit4:junit4] 2> 131657 T600 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[29 (-1438127644165013504)]} 0 1
[junit4:junit4] 2> 131657 T599 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10029 (-1438127644165013505)]} 0 1
[junit4:junit4] 2> 131667 T634 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[29 (-1438127644168159233)]} 0 5
[junit4:junit4] 2> 131667 T635 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10029 (-1438127644168159232)]} 0 5
[junit4:junit4] 2> 131668 T619 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[29 (-1438127644168159233)]} 0 9
[junit4:junit4] 2> 131668 T621 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10029 (-1438127644168159232)]} 0 9
[junit4:junit4] 2> 131672 T601 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10045 (1438127644179693568)]} 0 1
[junit4:junit4] 2> 131673 T598 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[45 (1438127644179693569)]} 0 2
[junit4:junit4] 2> 131679 T636 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10045 (1438127644183887872)]} 0 1
[junit4:junit4] 2> 131680 T637 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[45 (1438127644184936448)]} 0 1
[junit4:junit4] 2> 131680 T620 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10045 (1438127644183887872)]} 0 5
[junit4:junit4] 2> 131680 T618 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[45 (1438127644184936448)]} 0 5
[junit4:junit4] 2> 131706 T600 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[31 (-1438127644215345153)]} 0 1
[junit4:junit4] 2> 131706 T599 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10031 (-1438127644215345152)]} 0 1
[junit4:junit4] 2> 131712 T634 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[31 (-1438127644219539457)]} 0 1
[junit4:junit4] 2> 131712 T635 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10031 (-1438127644219539456)]} 0 1
[junit4:junit4] 2> 131713 T619 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[31 (-1438127644219539457)]} 0 5
[junit4:junit4] 2> 131713 T621 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10031 (-1438127644219539456)]} 0 5
[junit4:junit4] 2> 131717 T601 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[46 (1438127644226879488)]} 0 1
[junit4:junit4] 2> 131718 T598 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10046 (1438127644226879489)]} 0 2
[junit4:junit4] 2> 131726 T636 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[46 (1438127644232122368)]} 0 2
[junit4:junit4] 2> 131726 T637 C990 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10046 (1438127644232122369)]} 0 2
[junit4:junit4] 2> 131726 T620 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[46 (1438127644232122368)]} 0 6
[junit4:junit4] 2> 131727 T618 C991 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10046 (1438127644232122369)]} 0 6
[junit4:junit4] 2> 131801 T599 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[47 (1438127644314959872)]} 0 1
[junit4:junit4] 2> 131801 T600 C989 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10047 (1438127644314959873)]} 0 1
[junit4:junit4] 2> 131818 T606 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 131819 T606 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:35185_u_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35185/u_"}
[junit4:junit4] 2> ASYNC NEW_CORE C992 name=collection1 org.apache.solr.core.SolrCore@7c04c6ec url=http://127.0.0.1:35185/u_/collection1 node=127.0.0.1:35185_u_ C992_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:35185_u_, base_url=http://127.0.0.1:35185/u_}
[junit4:junit4] 2> 131819 T634 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10047 (1438127644319154176)]} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C993 name=collection1 org.apache.solr.core.SolrCore@4230efdd url=http://127.0.0.1:56556/u_/collection1 node=127.0.0.1:56556_u_ C993_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:56556_u_, base_url=http://127.0.0.1:56556/u_, leader=true}
[junit4:junit4] 2> 131820 T621 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10047 (1438127644319154176)]} 0 16
[junit4:junit4] 2> 131821 T636 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[47 (1438127644319154177)]} 0 2
[junit4:junit4] 2> 131822 T619 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[47 (1438127644319154177)]} 0 18
[junit4:junit4] 2> 131824 T606 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35185_u_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35185/u_"}
[junit4:junit4] 2> ASYNC NEW_CORE C994 name=collection1 org.apache.solr.core.SolrCore@658b0b44 url=http://127.0.0.1:34380/u_/collection1 node=127.0.0.1:34380_u_ C994_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:34380_u_, base_url=http://127.0.0.1:34380/u_, leader=true}
[junit4:junit4] 2> 131829 T601 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10032 (-1438127644345368576)]} 0 0
[junit4:junit4] 2> 131831 T598 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[32 (-1438127644347465728)]} 0 0
[junit4:junit4] 2> 131837 T637 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10032 (-1438127644349562880)]} 0 0
[junit4:junit4] 2> 131837 T635 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[32 (-1438127644349562881)]} 0 0
[junit4:junit4] 2> 131837 T620 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10032 (-1438127644349562880)]} 0 4
[junit4:junit4] 2> 131838 T618 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[32 (-1438127644349562881)]} 0 5
[junit4:junit4] 2> 131842 T599 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10048 (1438127644357951488)]} 0 2
[junit4:junit4] 2> 131842 T600 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[48 (1438127644357951489)]} 0 1
[junit4:junit4] 2> 131850 T634 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10048 (1438127644362145792)]} 0 1
[junit4:junit4] 2> 131851 T621 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10048 (1438127644362145792)]} 0 6
[junit4:junit4] 2> 131851 T636 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[48 (1438127644362145793)]} 0 2
[junit4:junit4] 2> 131852 T619 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[48 (1438127644362145793)]} 0 7
[junit4:junit4] 2> 131858 T601 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10033 (-1438127644375777280)]} 0 1
[junit4:junit4] 2> 131859 T598 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[33 (-1438127644376825856)]} 0 1
[junit4:junit4] 2> 131864 T637 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10033 (-1438127644378923008)]} 0 1
[junit4:junit4] 2> 131864 T635 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[33 (-1438127644379971584)]} 0 0
[junit4:junit4] 2> 131865 T620 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10033 (-1438127644378923008)]} 0 4
[junit4:junit4] 2> 131866 T618 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[33 (-1438127644379971584)]} 0 5
[junit4:junit4] 2> 131869 T600 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[49 (1438127644387311616)]} 0 1
[junit4:junit4] 2> 131869 T599 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10049 (1438127644386263040)]} 0 1
[junit4:junit4] 2> 131878 T634 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[49 (1438127644391505920)]} 0 2
[junit4:junit4] 2> 131879 T636 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10049 (1438127644391505921)]} 0 1
[junit4:junit4] 2> 131879 T621 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[49 (1438127644391505920)]} 0 6
[junit4:junit4] 2> 131879 T619 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10049 (1438127644391505921)]} 0 6
[junit4:junit4] 2> 131890 T598 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[34 (-1438127644409331713)]} 0 0
[junit4:junit4] 2> 131890 T601 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10034 (-1438127644409331712)]} 0 0
[junit4:junit4] 2> 131897 T635 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10034 (-1438127644413526017)]} 0 1
[junit4:junit4] 2> 131897 T618 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10034 (-1438127644413526017)]} 0 3
[junit4:junit4] 2> 131898 T637 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[34 (-1438127644413526016)]} 0 0
[junit4:junit4] 2> 131899 T620 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[34 (-1438127644413526016)]} 0 6
[junit4:junit4] 2> 131901 T600 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10050 (1438127644419817472)]} 0 1
[junit4:junit4] 2> 131903 T599 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[50 (1438127644421914624)]} 0 1
[junit4:junit4] 2> 131910 T634 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10050 (1438127644425060352)]} 0 1
[junit4:junit4] 2> 131911 T636 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[50 (1438127644426108928)]} 0 2
[junit4:junit4] 2> 131912 T621 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10050 (1438127644425060352)]} 0 7
[junit4:junit4] 2> 131912 T619 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[50 (1438127644426108928)]} 0 6
[junit4:junit4] 2> 131923 T605 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 131923 T625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 131923 T612 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 131923 T641 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 131990 T601 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10051 (1438127644513140737)]} 0 1
[junit4:junit4] 2> 131991 T598 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[51 (1438127644513140736)]} 0 2
[junit4:junit4] 2> C992_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:35185_u_, base_url=http://127.0.0.1:35185/u_}
[junit4:junit4] 2> 131999 T635 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10051 (1438127644518383617)]} 0 1
[junit4:junit4] 2> 132000 T637 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[51 (1438127644518383616)]} 0 2
[junit4:junit4] 2> 132000 T620 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10051 (1438127644518383617)]} 0 7
[junit4:junit4] 2> 132001 T618 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[51 (1438127644518383616)]} 0 8
[junit4:junit4] 2> 132092 T600 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[52 (1438127644620095488)]} 0 1
[junit4:junit4] 2> 132092 T599 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10052 (1438127644620095489)]} 0 1
[junit4:junit4] 2> 132100 T636 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10052 (1438127644625338368)]} 0 1
[junit4:junit4] 2> 132101 T619 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10052 (1438127644625338368)]} 0 5
[junit4:junit4] 2> 132103 T634 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[52 (1438127644628484096)]} 0 1
[junit4:junit4] 2> 132104 T621 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[52 (1438127644628484096)]} 0 6
[junit4:junit4] 2> 132182 T601 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10035 (-1438127644715515904)]} 0 0
[junit4:junit4] 2> 132184 T598 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[35 (-1438127644717613056)]} 0 0
[junit4:junit4] 2> 132187 T635 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10035 (-1438127644718661632)]} 0 0
[junit4:junit4] 2> 132188 T620 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10035 (-1438127644718661632)]} 0 3
[junit4:junit4] 2> 132190 T637 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[35 (-1438127644720758784)]} 0 1
[junit4:junit4] 2> 132190 T618 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[35 (-1438127644720758784)]} 0 3
[junit4:junit4] 2> 132192 T599 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10053 (1438127644723904512)]} 0 2
[junit4:junit4] 2> 132194 T600 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[53 (1438127644727050240)]} 0 1
[junit4:junit4] 2> 132199 T636 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10053 (1438127644729147392)]} 0 1
[junit4:junit4] 2> 132200 T619 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10053 (1438127644729147392)]} 0 6
[junit4:junit4] 2> 132202 T634 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[53 (1438127644732293120)]} 0 1
[junit4:junit4] 2> 132203 T621 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[53 (1438127644732293120)]} 0 6
[junit4:junit4] 2> 132250 T601 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10042 (-1438127644786819072)]} 0 0
[junit4:junit4] 2> 132253 T599 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[42 (-1438127644789964800)]} 0 0
[junit4:junit4] 2> 132256 T635 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10042 (-1438127644789964800)]} 0 1
[junit4:junit4] 2> 132256 T620 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10042 (-1438127644789964800)]} 0 3
[junit4:junit4] 2> 132259 T636 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[42 (-1438127644793110528)]} 0 1
[junit4:junit4] 2> 132259 T618 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[42 (-1438127644793110528)]} 0 3
[junit4:junit4] 2> 132261 T600 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10054 (1438127644796256256)]} 0 2
[junit4:junit4] 2> 132264 T598 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[54 (1438127644800450560)]} 0 2
[junit4:junit4] 2> 132268 T634 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10054 (1438127644800450560)]} 0 1
[junit4:junit4] 2> 132269 T619 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10054 (1438127644800450560)]} 0 6
[junit4:junit4] 2> 132272 T637 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[54 (1438127644804644864)]} 0 1
[junit4:junit4] 2> 132272 T621 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[54 (1438127644804644864)]} 0 5
[junit4:junit4] 2> 132361 T599 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10044 (-1438127644903211008)]} 0 0
[junit4:junit4] 2> 132364 T600 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[44 (-1438127644906356736)]} 0 0
[junit4:junit4] 2> 132366 T635 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10044 (-1438127644906356736)]} 0 0
[junit4:junit4] 2> 132367 T620 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10044 (-1438127644906356736)]} 0 3
[junit4:junit4] 2> 132369 T636 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[44 (-1438127644909502464)]} 0 0
[junit4:junit4] 2> 132370 T618 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[44 (-1438127644909502464)]} 0 3
[junit4:junit4] 2> 132371 T601 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10055 (1438127644912648192)]} 0 2
[junit4:junit4] 2> 132373 T598 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[55 (1438127644914745344)]} 0 1
[junit4:junit4] 2> 132379 T634 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10055 (1438127644916842496)]} 0 1
[junit4:junit4] 2> 132380 T619 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10055 (1438127644916842496)]} 0 7
[junit4:junit4] 2> 132384 T637 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[55 (1438127644919988224)]} 0 2
[junit4:junit4] 2> 132384 T621 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[55 (1438127644919988224)]} 0 8
[junit4:junit4] 2> 132453 T599 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10048 (-1438127644999680000)]} 0 1
[junit4:junit4] 2> 132456 T600 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[48 (-1438127645002825728)]} 0 0
[junit4:junit4] 2> 132458 T635 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10048 (-1438127645001777152)]} 0 1
[junit4:junit4] 2> 132458 T620 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10048 (-1438127645001777152)]} 0 3
[junit4:junit4] 2> 132461 T636 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[48 (-1438127645004922880)]} 0 0
[junit4:junit4] 2> 132462 T619 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[48 (-1438127645004922880)]} 0 4
[junit4:junit4] 2> 132463 T601 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10056 (1438127645008068608)]} 0 2
[junit4:junit4] 2> 132467 T598 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[56 (1438127645012262912)]} 0 2
[junit4:junit4] 2> 132471 T634 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10056 (1438127645013311488)]} 0 2
[junit4:junit4] 2> 132472 T618 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10056 (1438127645013311488)]} 0 6
[junit4:junit4] 2> 132475 T637 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[56 (1438127645017505792)]} 0 2
[junit4:junit4] 2> 132476 T621 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[56 (1438127645017505792)]} 0 6
[junit4:junit4] 2> 132522 T599 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10053 (-1438127645072031744)]} 0 1
[junit4:junit4] 2> 132525 T600 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[53 (-1438127645075177472)]} 0 0
[junit4:junit4] 2> 132528 T636 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10053 (-1438127645075177472)]} 0 1
[junit4:junit4] 2> 132528 T620 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10053 (-1438127645075177472)]} 0 3
[junit4:junit4] 2> 132532 T634 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[53 (-1438127645079371776)]} 0 1
[junit4:junit4] 2> 132533 T601 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10057 (1438127645081468928)]} 0 2
[junit4:junit4] 2> 132533 T619 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[53 (-1438127645079371776)]} 0 5
[junit4:junit4] 2> 132536 T598 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[57 (1438127645085663232)]} 0 1
[junit4:junit4] 2> 132540 T635 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10057 (1438127645085663232)]} 0 2
[junit4:junit4] 2> 132540 T618 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10057 (1438127645085663232)]} 0 5
[junit4:junit4] 2> 132543 T637 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[57 (1438127645089857536)]} 0 1
[junit4:junit4] 2> 132544 T621 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[57 (1438127645089857536)]} 0 5
[junit4:junit4] 2> 132591 T599 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10054 (-1438127645144383488)]} 0 0
[junit4:junit4] 2> 132595 T600 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[54 (-1438127645148577792)]} 0 1
[junit4:junit4] 2> 132596 T636 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10054 (-1438127645147529216)]} 0 0
[junit4:junit4] 2> 132597 T620 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[10054 (-1438127645147529216)]} 0 3
[junit4:junit4] 2> 132600 T634 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[54 (-1438127645150674944)]} 0 0
[junit4:junit4] 2> 132600 T619 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {delete=[54 (-1438127645150674944)]} 0 3
[junit4:junit4] 2> 132601 T601 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10058 (1438127645153820672)]} 0 2
[junit4:junit4] 2> 132604 T598 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[58 (1438127645156966400)]} 0 1
[junit4:junit4] 2> 132608 T635 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10058 (1438127645158014976)]} 0 1
[junit4:junit4] 2> 132609 T618 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10058 (1438127645158014976)]} 0 6
[junit4:junit4] 2> 132611 T637 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[58 (1438127645161160704)]} 0 1
[junit4:junit4] 2> 132612 T621 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[58 (1438127645161160704)]} 0 5
[junit4:junit4] 2> 132691 T599 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10059 (1438127645248192512)]} 0 2
[junit4:junit4] 2> 132693 T601 C994 P34380 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[59 (1438127645250289664)]} 0 1
[junit4:junit4] 2> 132697 T636 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10059 (1438127645251338240)]} 0 1
[junit4:junit4] 2> 132698 T620 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[10059 (1438127645251338240)]} 0 5
[junit4:junit4] 2> 132700 T634 C992 P35185 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={distrib.from=http://127.0.0.1:56556/u_/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[59 (1438127645254483968)]} 0 2
[junit4:junit4] 2> 132700 T619 C993 P56556 oasup.LogUpdateProcessor.finish [collection1] webapp=/u_ path=/update params={wt=javabin&version=2} {add=[59 (1438127645254483968)]} 0 4
[junit4:junit4] 2> 132740 T600 C9
[...truncated too long message...]
java:294)
[junit4:junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4:junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4:junit4] 2> ... 5 more
[junit4:junit4] 2> Caused by: java.net.ConnectException: Connection refused
[junit4:junit4] 2> at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4:junit4] 2> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4:junit4] 2> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4:junit4] 2> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4:junit4] 2> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
[junit4:junit4] 2> at java.net.Socket.connect(Socket.java:579)
[junit4:junit4] 2> at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4:junit4] 2> at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4:junit4] 2> ... 12 more
[junit4:junit4] 2>
[junit4:junit4] 2> 253699 T665 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4] 2> 253699 T665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 253702 T665 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 253704 T665 oasc.Overseer.start Overseer (id=89882976948977674-127.0.0.1:35185_u_-n_0000000003) starting
[junit4:junit4] 2> 253709 T680 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 253710 T679 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 253712 T679 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 253713 T679 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56556_u_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56556/u_"}
[junit4:junit4] 2> 253729 T665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 253739 T583 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 35185
[junit4:junit4] 2> 253740 T583 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=658076417
[junit4:junit4] 2> 254742 T583 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 254743 T583 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 254744 T583 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@3d30ae63
[junit4:junit4] 2> 254750 T583 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=43,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=72,cumulative_deletesById=36,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 254751 T583 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 254751 T583 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 254752 T583 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 254753 T583 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 254754 T583 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 254754 T583 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2;done=false>>]
[junit4:junit4] 2> 254754 T583 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2
[junit4:junit4] 2> 254755 T583 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2/index;done=false>>]
[junit4:junit4] 2> 254755 T583 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371505385572/jetty2/index
[junit4:junit4] 2> 254756 T679 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89882976948977674-127.0.0.1:35185_u_-n_0000000003) am no longer a leader.
[junit4:junit4] 2> 254778 T583 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/u_,null}
[junit4:junit4] 2> 254834 T583 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 254836 T583 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:34322 34322
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.method=testDistribSearch -Dtests.seed=B7A833C1E8811B6C -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=tr_TR -Dtests.timezone=SystemV/YST9 -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 136s J0 | RecoveryZkTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: expected:<614> but was:<72>
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([B7A833C1E8811B6C:364EBDD99FDE7B50]:0)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 254882 T583 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 135615 T582 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 256057 T665 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 256058 T665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 256058 T665 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=tr_TR, timezone=SystemV/YST9
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=8,threads=3,free=78154736,total=158711808
[junit4:junit4] 2> NOTE: All tests run in this JVM: [SpellingQueryConverterTest, CachingDirectoryFactoryTest, ResourceLoaderTest, TermsComponentTest, CoreAdminHandlerTest, QueryParsingTest, TermVectorComponentTest, TestGroupingSearch, RAMDirectoryFactoryTest, TestRealTimeGet, TestUniqueKeyFieldResource, SliceStateUpdateTest, DistributedQueryElevationComponentTest, TestCloudManagedSchema, TestExtendedDismaxParser, SoftAutoCommitTest, TestDocSet, DirectUpdateHandlerOptimizeTest, DistributedTermsComponentTest, TestSearchPerf, RecoveryZkTest]
[junit4:junit4] Completed on J0 in 136.89s, 1 test, 1 failure <<< FAILURES!
[...truncated 731 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1249 tests, 1 failure, 13 ignored (7 assumptions)
Total time: 36 minutes 47 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_21) - Build #
6181 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6181/
Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseParallelGC
2 tests failed.
REGRESSION: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
expected:<588> but was:<32>
Stack Trace:
java.lang.AssertionError: expected:<588> but was:<32>
at __randomizedtesting.SeedInfo.seed([132BBD2C8A8DA51:80D435CABFF7BA6D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
REGRESSION: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:50139/_wvf/collection1lastClient and got 252 from http://127.0.0.1:56425/_wvf/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:50139/_wvf/collection1lastClient and got 252 from http://127.0.0.1:56425/_wvf/collection1
at __randomizedtesting.SeedInfo.seed([132BBD2C8A8DA51:80D435CABFF7BA6D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9616 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4] 2> 520581 T2040 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_wvf/
[junit4:junit4] 2> 520585 T2040 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SyncSliceTest-1371500907603
[junit4:junit4] 2> 520585 T2040 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 520586 T2041 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 520686 T2040 oasc.ZkTestServer.run start zk server on port:45035
[junit4:junit4] 2> 520687 T2040 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 520797 T2047 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61d01e02 name:ZooKeeperConnection Watcher:127.0.0.1:45035 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 520797 T2040 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 520798 T2040 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 520802 T2040 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 520803 T2049 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@189481d1 name:ZooKeeperConnection Watcher:127.0.0.1:45035/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 520804 T2040 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 520804 T2040 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 520809 T2040 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 520812 T2040 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 520813 T2040 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 520815 T2040 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 520815 T2040 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 520818 T2040 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 520819 T2040 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 520821 T2040 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 520821 T2040 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 520823 T2040 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 520823 T2040 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 520825 T2040 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 520826 T2040 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 520827 T2040 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 520828 T2040 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 520829 T2040 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 520830 T2040 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 520831 T2040 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 520832 T2040 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 520833 T2040 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 520834 T2040 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 520893 T2040 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 520896 T2040 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36298
[junit4:junit4] 2> 520896 T2040 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 520896 T2040 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 520897 T2040 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371500907854
[junit4:junit4] 2> 520897 T2040 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371500907854/solr.xml
[junit4:junit4] 2> 520897 T2040 oasc.CoreContainer.<init> New CoreContainer 1741711994
[junit4:junit4] 2> 520897 T2040 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371500907854/'
[junit4:junit4] 2> 520898 T2040 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371500907854/'
[junit4:junit4] 2> 520936 T2040 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 520936 T2040 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 520937 T2040 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 520937 T2040 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 520937 T2040 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 520937 T2040 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 520938 T2040 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 520938 T2040 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 520938 T2040 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 520939 T2040 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 520942 T2040 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 520942 T2040 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45035/solr
[junit4:junit4] 2> 520942 T2040 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 520943 T2040 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 520944 T2060 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@213213df name:ZooKeeperConnection Watcher:127.0.0.1:45035 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 520945 T2040 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 520946 T2040 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 520949 T2040 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 520950 T2062 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f50dac9 name:ZooKeeperConnection Watcher:127.0.0.1:45035/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 520950 T2040 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 520952 T2040 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 520955 T2040 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 520959 T2040 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 520961 T2040 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36298__wvf
[junit4:junit4] 2> 520962 T2040 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36298__wvf
[junit4:junit4] 2> 520965 T2040 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 520970 T2040 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 520971 T2040 oasc.Overseer.start Overseer (id=89882683480801283-127.0.0.1:36298__wvf-n_0000000000) starting
[junit4:junit4] 2> 520974 T2040 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 520976 T2064 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 520976 T2040 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 520978 T2040 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 520979 T2040 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 520981 T2063 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 520983 T2065 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 520983 T2065 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 522484 T2063 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 522485 T2063 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:36298__wvf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:36298/_wvf"}
[junit4:junit4] 2> 522485 T2063 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 522485 T2063 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 522490 T2062 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 522984 T2065 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371500907854/collection1
[junit4:junit4] 2> 522985 T2065 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 522985 T2065 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 522985 T2065 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 522987 T2065 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371500907854/collection1/'
[junit4:junit4] 2> 522988 T2065 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-controljetty-1371500907854/collection1/lib/README' to classloader
[junit4:junit4] 2> 522988 T2065 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-controljetty-1371500907854/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 523029 T2065 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 523078 T2065 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 523079 T2065 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 523085 T2065 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 523523 T2065 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 523532 T2065 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 523535 T2065 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 523553 T2065 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 523558 T2065 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 523563 T2065 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 523565 T2065 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 523565 T2065 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 523565 T2065 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 523566 T2065 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 523567 T2065 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 523567 T2065 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 523568 T2065 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371500907854/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371500907603/control/data/
[junit4:junit4] 2> 523568 T2065 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@39061e68
[junit4:junit4] 2> 523570 T2065 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 523570 T2065 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/control/data
[junit4:junit4] 2> 523570 T2065 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371500907603/control/data/index/
[junit4:junit4] 2> 523571 T2065 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371500907603/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 523571 T2065 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/control/data/index
[junit4:junit4] 2> 523578 T2065 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 523579 T2065 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 523581 T2065 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 523581 T2065 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 523582 T2065 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 523583 T2065 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 523583 T2065 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 523583 T2065 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 523583 T2065 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 523584 T2065 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 523584 T2065 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 523588 T2065 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 523593 T2065 oass.SolrIndexSearcher.<init> Opening Searcher@1c1c6da8 main
[junit4:junit4] 2> 523594 T2065 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 523594 T2065 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 523599 T2066 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c1c6da8 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 523600 T2065 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 523600 T2065 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36298/_wvf collection:control_collection shard:shard1
[junit4:junit4] 2> 523601 T2065 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 523616 T2065 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 523619 T2065 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 523619 T2065 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 523620 T2065 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:36298/_wvf/collection1/
[junit4:junit4] 2> 523620 T2065 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 523620 T2065 oasc.SyncStrategy.syncToMe http://127.0.0.1:36298/_wvf/collection1/ has no replicas
[junit4:junit4] 2> 523620 T2065 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:36298/_wvf/collection1/
[junit4:junit4] 2> 523621 T2065 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 523994 T2063 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 524003 T2062 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 524031 T2065 oasc.ZkController.register We are http://127.0.0.1:36298/_wvf/collection1/ and leader is http://127.0.0.1:36298/_wvf/collection1/
[junit4:junit4] 2> 524031 T2065 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36298/_wvf
[junit4:junit4] 2> 524031 T2065 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 524032 T2065 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 524032 T2065 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 524034 T2065 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 524035 T2040 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 524036 T2040 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 524036 T2040 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 524040 T2040 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 524052 T2040 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 524052 T2069 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ca5c369 name:ZooKeeperConnection Watcher:127.0.0.1:45035/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 524052 T2040 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 524055 T2040 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 524064 T2040 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 524205 T2040 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 524208 T2040 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50139
[junit4:junit4] 2> 524209 T2040 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 524209 T2040 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 524210 T2040 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371500911083
[junit4:junit4] 2> 524210 T2040 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371500911083/solr.xml
[junit4:junit4] 2> 524211 T2040 oasc.CoreContainer.<init> New CoreContainer 1078407628
[junit4:junit4] 2> 524211 T2040 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371500911083/'
[junit4:junit4] 2> 524212 T2040 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371500911083/'
[junit4:junit4] 2> 524282 T2040 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 524283 T2040 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 524283 T2040 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 524284 T2040 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 524284 T2040 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 524284 T2040 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 524285 T2040 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 524285 T2040 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 524285 T2040 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 524286 T2040 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 524291 T2040 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 524291 T2040 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45035/solr
[junit4:junit4] 2> 524292 T2040 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 524293 T2040 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 524294 T2082 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@245c4071 name:ZooKeeperConnection Watcher:127.0.0.1:45035 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 524295 T2040 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 524298 T2040 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 524302 T2040 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 524304 T2084 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18b74568 name:ZooKeeperConnection Watcher:127.0.0.1:45035/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 524304 T2040 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 524311 T2040 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 525315 T2040 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50139__wvf
[junit4:junit4] 2> 525316 T2040 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50139__wvf
[junit4:junit4] 2> 525320 T2062 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 525320 T2069 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 525320 T2084 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 525320 T2062 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 525326 T2085 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 525327 T2085 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 525510 T2063 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 525511 T2063 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:36298__wvf_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:36298__wvf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:36298/_wvf"}
[junit4:junit4] 2> 525515 T2063 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50139__wvf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50139/_wvf"}
[junit4:junit4] 2> 525516 T2063 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 525516 T2063 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 525521 T2084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 525521 T2069 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 525523 T2062 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 526328 T2085 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371500911083/collection1
[junit4:junit4] 2> 526328 T2085 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 526329 T2085 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 526330 T2085 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 526331 T2085 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371500911083/collection1/'
[junit4:junit4] 2> 526331 T2085 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty1-1371500911083/collection1/lib/README' to classloader
[junit4:junit4] 2> 526332 T2085 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty1-1371500911083/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 526367 T2085 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 526422 T2085 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 526423 T2085 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 526428 T2085 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 526944 T2085 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 526952 T2085 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 526956 T2085 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 526970 T2085 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 526977 T2085 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 526984 T2085 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 526986 T2085 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 526987 T2085 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 526987 T2085 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 526988 T2085 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 526988 T2085 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 526988 T2085 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 526989 T2085 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371500911083/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/
[junit4:junit4] 2> 526989 T2085 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@39061e68
[junit4:junit4] 2> 526990 T2085 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 526990 T2085 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1
[junit4:junit4] 2> 526991 T2085 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index/
[junit4:junit4] 2> 526991 T2085 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 526991 T2085 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index
[junit4:junit4] 2> 526996 T2085 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 526997 T2085 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 526999 T2085 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 527000 T2085 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 527000 T2085 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 527001 T2085 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 527001 T2085 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 527001 T2085 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 527002 T2085 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 527002 T2085 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 527003 T2085 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 527007 T2085 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 527012 T2085 oass.SolrIndexSearcher.<init> Opening Searcher@3ca6b2e9 main
[junit4:junit4] 2> 527012 T2085 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 527013 T2085 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 527020 T2085 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 527020 T2085 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50139/_wvf collection:collection1 shard:shard1
[junit4:junit4] 2> 527021 T2085 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 527022 T2086 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3ca6b2e9 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 527028 T2085 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 527030 T2085 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 527030 T2085 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 527030 T2085 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50139/_wvf/collection1/
[junit4:junit4] 2> 527031 T2085 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 527031 T2085 oasc.SyncStrategy.syncToMe http://127.0.0.1:50139/_wvf/collection1/ has no replicas
[junit4:junit4] 2> 527031 T2085 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50139/_wvf/collection1/
[junit4:junit4] 2> 527031 T2085 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 528530 T2063 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 528537 T2084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 528537 T2069 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 528537 T2062 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 528544 T2085 oasc.ZkController.register We are http://127.0.0.1:50139/_wvf/collection1/ and leader is http://127.0.0.1:50139/_wvf/collection1/
[junit4:junit4] 2> 528544 T2085 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50139/_wvf
[junit4:junit4] 2> 528544 T2085 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 528545 T2085 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 528545 T2085 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 528546 T2085 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 528547 T2040 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 528548 T2040 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 528548 T2040 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 528638 T2040 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 528640 T2040 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39311
[junit4:junit4] 2> 528640 T2040 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 528640 T2040 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 528641 T2040 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371500915570
[junit4:junit4] 2> 528641 T2040 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371500915570/solr.xml
[junit4:junit4] 2> 528642 T2040 oasc.CoreContainer.<init> New CoreContainer 2142018818
[junit4:junit4] 2> 528642 T2040 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371500915570/'
[junit4:junit4] 2> 528642 T2040 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371500915570/'
[junit4:junit4] 2> 528701 T2040 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 528702 T2040 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 528702 T2040 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 528702 T2040 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 528703 T2040 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 528703 T2040 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 528704 T2040 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 528704 T2040 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 528704 T2040 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 528705 T2040 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 528708 T2040 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 528709 T2040 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45035/solr
[junit4:junit4] 2> 528709 T2040 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 528710 T2040 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 528712 T2098 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c8b7814 name:ZooKeeperConnection Watcher:127.0.0.1:45035 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 528713 T2040 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 528715 T2040 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 528718 T2040 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 528722 T2100 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5dee9293 name:ZooKeeperConnection Watcher:127.0.0.1:45035/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 528722 T2040 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 528726 T2040 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 529731 T2040 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39311__wvf
[junit4:junit4] 2> 529732 T2040 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39311__wvf
[junit4:junit4] 2> 529735 T2069 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 529735 T2062 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 529735 T2084 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 529735 T2062 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 529736 T2100 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 529736 T2084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 529736 T2069 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 529741 T2101 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 529742 T2101 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 530042 T2063 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 530043 T2063 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:50139__wvf_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50139__wvf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50139/_wvf"}
[junit4:junit4] 2> 530046 T2063 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39311__wvf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39311/_wvf"}
[junit4:junit4] 2> 530046 T2063 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 530046 T2063 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 530049 T2100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 530049 T2069 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 530049 T2084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 530049 T2062 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 530743 T2101 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371500915570/collection1
[junit4:junit4] 2> 530744 T2101 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 530744 T2101 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 530744 T2101 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 530745 T2101 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371500915570/collection1/'
[junit4:junit4] 2> 530746 T2101 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty2-1371500915570/collection1/lib/README' to classloader
[junit4:junit4] 2> 530746 T2101 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty2-1371500915570/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 530774 T2101 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 530807 T2101 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 530809 T2101 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 530812 T2101 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 531155 T2101 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 531162 T2101 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 531164 T2101 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 531177 T2101 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 531180 T2101 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 531183 T2101 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 531184 T2101 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 531185 T2101 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 531185 T2101 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 531186 T2101 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 531186 T2101 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 531187 T2101 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 531187 T2101 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371500915570/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/
[junit4:junit4] 2> 531187 T2101 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@39061e68
[junit4:junit4] 2> 531187 T2101 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 531188 T2101 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2
[junit4:junit4] 2> 531188 T2101 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index/
[junit4:junit4] 2> 531188 T2101 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 531189 T2101 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index
[junit4:junit4] 2> 531191 T2101 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 531192 T2101 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 531194 T2101 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 531194 T2101 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 531195 T2101 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 531195 T2101 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 531195 T2101 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 531196 T2101 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 531196 T2101 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 531196 T2101 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 531197 T2101 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 531199 T2101 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 531202 T2101 oass.SolrIndexSearcher.<init> Opening Searcher@682103a7 main
[junit4:junit4] 2> 531202 T2101 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 531203 T2101 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 531207 T2102 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@682103a7 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 531209 T2101 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 531209 T2101 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39311/_wvf collection:collection1 shard:shard1
[junit4:junit4] 2> 531211 T2101 oasc.ZkController.register We are http://127.0.0.1:39311/_wvf/collection1/ and leader is http://127.0.0.1:50139/_wvf/collection1/
[junit4:junit4] 2> 531212 T2101 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39311/_wvf
[junit4:junit4] 2> 531212 T2101 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 531212 T2101 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C366 name=collection1 org.apache.solr.core.SolrCore@497bb53c url=http://127.0.0.1:39311/_wvf/collection1 node=127.0.0.1:39311__wvf C366_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:39311__wvf, base_url=http://127.0.0.1:39311/_wvf}
[junit4:junit4] 2> 531212 T2103 C366 P39311 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 531213 T2103 C366 P39311 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 531213 T2101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 531213 T2103 C366 P39311 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 531214 T2103 C366 P39311 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 531214 T2040 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 531215 T2040 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 531215 T2103 C366 P39311 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 531215 T2040 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 531220 T2077 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:39311__wvf_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 531320 T2040 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 531322 T2040 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56425
[junit4:junit4] 2> 531323 T2040 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 531324 T2040 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 531324 T2040 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371500918236
[junit4:junit4] 2> 531325 T2040 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371500918236/solr.xml
[junit4:junit4] 2> 531325 T2040 oasc.CoreContainer.<init> New CoreContainer 1972332217
[junit4:junit4] 2> 531326 T2040 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371500918236/'
[junit4:junit4] 2> 531326 T2040 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371500918236/'
[junit4:junit4] 2> 531389 T2040 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 531390 T2040 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 531390 T2040 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 531390 T2040 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 531391 T2040 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 531391 T2040 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 531391 T2040 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 531392 T2040 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 531392 T2040 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 531392 T2040 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 531396 T2040 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 531397 T2040 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45035/solr
[junit4:junit4] 2> 531397 T2040 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 531399 T2040 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 531400 T2115 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@364d50af name:ZooKeeperConnection Watcher:127.0.0.1:45035 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 531401 T2040 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 531403 T2040 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 531406 T2040 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 531408 T2117 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ec0863e name:ZooKeeperConnection Watcher:127.0.0.1:45035/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 531408 T2040 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 531413 T2040 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 531556 T2063 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 531557 T2063 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:39311__wvf_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39311__wvf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39311/_wvf"}
[junit4:junit4] 2> 531560 T2117 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 531560 T2069 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 531560 T2084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 531560 T2062 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 531560 T2100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 532220 T2077 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:39311__wvf_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 532221 T2077 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:39311__wvf_collection1&state=recovering&nodeName=127.0.0.1:39311__wvf&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 532418 T2040 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56425__wvf
[junit4:junit4] 2> 532419 T2040 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56425__wvf
[junit4:junit4] 2> 532421 T2117 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 532421 T2069 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 532422 T2062 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 532422 T2100 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 532422 T2084 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 532422 T2100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 532422 T2062 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 532422 T2084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 532423 T2117 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 532424 T2069 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 532428 T2118 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 532429 T2118 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 533065 T2063 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 533065 T2063 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56425__wvf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56425/_wvf"}
[junit4:junit4] 2> 533066 T2063 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 533066 T2063 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 533071 T2100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 533071 T2062 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 533071 T2069 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 533071 T2117 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 533073 T2084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 533430 T2118 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371500918236/collection1
[junit4:junit4] 2> 533431 T2118 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 533431 T2118 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 533431 T2118 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 533433 T2118 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371500918236/collection1/'
[junit4:junit4] 2> 533433 T2118 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty3-1371500918236/collection1/lib/README' to classloader
[junit4:junit4] 2> 533433 T2118 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty3-1371500918236/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 533462 T2118 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 533495 T2118 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 533496 T2118 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 533500 T2118 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 533839 T2118 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 533845 T2118 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 533848 T2118 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 533859 T2118 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 533863 T2118 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 533866 T2118 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 533867 T2118 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 533868 T2118 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 533868 T2118 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 533869 T2118 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 533869 T2118 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 533869 T2118 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 533870 T2118 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371500918236/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/
[junit4:junit4] 2> 533870 T2118 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@39061e68
[junit4:junit4] 2> 533870 T2118 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 533871 T2118 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3
[junit4:junit4] 2> 533871 T2118 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index/
[junit4:junit4] 2> 533871 T2118 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 533872 T2118 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index
[junit4:junit4] 2> 533876 T2118 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 533876 T2118 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 533878 T2118 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 533879 T2118 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 533879 T2118 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 533880 T2118 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 533880 T2118 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 533881 T2118 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 533881 T2118 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 533881 T2118 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 533882 T2118 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 533886 T2118 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 533889 T2118 oass.SolrIndexSearcher.<init> Opening Searcher@7b5b086 main
[junit4:junit4] 2> 533890 T2118 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 533891 T2118 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 533895 T2119 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7b5b086 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 533897 T2118 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 533897 T2118 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56425/_wvf collection:collection1 shard:shard1
[junit4:junit4] 2> 533900 T2118 oasc.ZkController.register We are http://127.0.0.1:56425/_wvf/collection1/ and leader is http://127.0.0.1:50139/_wvf/collection1/
[junit4:junit4] 2> 533900 T2118 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56425/_wvf
[junit4:junit4] 2> 533900 T2118 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 533900 T2118 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C367 name=collection1 org.apache.solr.core.SolrCore@f65639e url=http://127.0.0.1:56425/_wvf/collection1 node=127.0.0.1:56425__wvf C367_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:56425__wvf, base_url=http://127.0.0.1:56425/_wvf}
[junit4:junit4] 2> 533901 T2120 C367 P56425 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 533902 T2118 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 533902 T2120 C367 P56425 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 533903 T2120 C367 P56425 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 533903 T2120 C367 P56425 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 533903 T2040 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 533904 T2040 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 533904 T2120 C367 P56425 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 533904 T2040 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 533909 T2078 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:56425__wvf_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 533978 T2040 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 533980 T2040 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54789
[junit4:junit4] 2> 533981 T2040 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 533982 T2040 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 533982 T2040 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371500920926
[junit4:junit4] 2> 533983 T2040 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371500920926/solr.xml
[junit4:junit4] 2> 533983 T2040 oasc.CoreContainer.<init> New CoreContainer 209140749
[junit4:junit4] 2> 533983 T2040 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371500920926/'
[junit4:junit4] 2> 533983 T2040 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371500920926/'
[junit4:junit4] 2> 534035 T2040 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 534035 T2040 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 534036 T2040 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 534036 T2040 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 534037 T2040 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 534037 T2040 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 534037 T2040 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 534037 T2040 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 534038 T2040 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 534038 T2040 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 534041 T2040 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 534042 T2040 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:45035/solr
[junit4:junit4] 2> 534042 T2040 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 534043 T2040 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 534045 T2132 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33c078fa name:ZooKeeperConnection Watcher:127.0.0.1:45035 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 534047 T2040 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 534049 T2040 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 534052 T2040 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 534054 T2134 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1dfafb44 name:ZooKeeperConnection Watcher:127.0.0.1:45035/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 534054 T2040 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 534059 T2040 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C368 name=collection1 org.apache.solr.core.SolrCore@497bb53c url=http://127.0.0.1:39311/_wvf/collection1 node=127.0.0.1:39311__wvf C368_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:39311__wvf, base_url=http://127.0.0.1:39311/_wvf}
[junit4:junit4] 2> 534222 T2103 C368 P39311 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:50139/_wvf/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 534222 T2103 C368 P39311 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39311/_wvf START replicas=[http://127.0.0.1:50139/_wvf/collection1/] nUpdates=100
[junit4:junit4] 2> 534222 T2103 C368 P39311 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 534223 T2103 C368 P39311 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 534223 T2103 C368 P39311 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 534223 T2103 C368 P39311 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 534223 T2103 C368 P39311 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 534224 T2103 C368 P39311 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:50139/_wvf/collection1/. core=collection1
[junit4:junit4] 2> 534224 T2103 C368 P39311 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C369 name=collection1 org.apache.solr.core.SolrCore@1dd44625 url=http://127.0.0.1:50139/_wvf/collection1 node=127.0.0.1:50139__wvf C369_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:50139__wvf, base_url=http://127.0.0.1:50139/_wvf, leader=true}
[junit4:junit4] 2> 534230 T2079 C369 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 534232 T2074 C369 P50139 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 534235 T2074 C369 P50139 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 534235 T2074 C369 P50139 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 534246 T2074 C369 P50139 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 534247 T2074 C369 P50139 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 534248 T2074 C369 P50139 oass.SolrIndexSearcher.<init> Opening Searcher@2b342ecf realtime
[junit4:junit4] 2> 534248 T2074 C369 P50139 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 534248 T2074 C369 P50139 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 16
[junit4:junit4] 2> 534249 T2103 C368 P39311 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 534250 T2103 C368 P39311 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 534252 T2076 C369 P50139 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 534252 T2076 C369 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 534253 T2103 C368 P39311 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 534253 T2103 C368 P39311 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 534253 T2103 C368 P39311 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 534256 T2079 C369 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 534256 T2103 C368 P39311 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 534257 T2103 C368 P39311 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index.20130618012841275
[junit4:junit4] 2> 534258 T2103 C368 P39311 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index.20130618012841275 lockFactory=org.apache.lucene.store.NativeFSLockFactory@26edc824 fullCopy=false
[junit4:junit4] 2> 534260 T2074 C369 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 534262 T2103 C368 P39311 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 534267 T2103 C368 P39311 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 534267 T2103 C368 P39311 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 534270 T2103 C368 P39311 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 534270 T2103 C368 P39311 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 534271 T2103 C368 P39311 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 534271 T2103 C368 P39311 oass.SolrIndexSearcher.<init> Opening Searcher@418b4dd main
[junit4:junit4] 2> 534272 T2102 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@418b4dd main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 534273 T2103 C368 P39311 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index.20130618012841275 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index.20130618012841275;done=true>>]
[junit4:junit4] 2> 534274 T2103 C368 P39311 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index.20130618012841275
[junit4:junit4] 2> 534274 T2103 C368 P39311 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index.20130618012841275
[junit4:junit4] 2> 534275 T2103 C368 P39311 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 534275 T2103 C368 P39311 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 534275 T2103 C368 P39311 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 534276 T2103 C368 P39311 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 534278 T2103 C368 P39311 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 534575 T2063 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 534576 T2063 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:56425__wvf_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56425__wvf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56425/_wvf"}
[junit4:junit4] 2> 534579 T2063 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:39311__wvf_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39311__wvf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39311/_wvf"}
[junit4:junit4] 2> 534599 T2084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 534600 T2062 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 534600 T2100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 534600 T2117 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 534599 T2134 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 534603 T2069 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 534910 T2078 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:56425__wvf_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 534910 T2078 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:56425__wvf_collection1&state=recovering&nodeName=127.0.0.1:56425__wvf&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 535064 T2040 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54789__wvf
[junit4:junit4] 2> 535065 T2040 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54789__wvf
[junit4:junit4] 2> 535067 T2117 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 535067 T2069 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 535068 T2134 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 535068 T2100 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 535068 T2100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 535068 T2084 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 535069 T2062 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 535069 T2084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 535069 T2062 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 535070 T2069 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 535070 T2117 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 535071 T2134 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 535076 T2137 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 535077 T2137 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 536107 T2063 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 536107 T2063 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:54789__wvf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54789/_wvf"}
[junit4:junit4] 2> 536108 T2063 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 536108 T2063 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 536112 T2100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 536112 T2134 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 536112 T2117 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 536112 T2084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 536112 T2062 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 536112 T2069 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> C367_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:56425__wvf, base_url=http://127.0.0.1:56425/_wvf}
[junit4:junit4] 2> 536911 T2120 C367 P56425 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:50139/_wvf/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 536911 T2120 C367 P56425 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56425/_wvf START replicas=[http://127.0.0.1:50139/_wvf/collection1/] nUpdates=100
[junit4:junit4] 2> 536912 T2120 C367 P56425 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 536912 T2120 C367 P56425 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 536913 T2120 C367 P56425 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 536913 T2120 C367 P56425 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 536913 T2120 C367 P56425 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 536913 T2120 C367 P56425 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:50139/_wvf/collection1/. core=collection1
[junit4:junit4] 2> 536913 T2120 C367 P56425 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 536914 T2074 C369 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 536918 T2078 C369 P50139 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 536926 T2078 C369 P50139 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> 536927 T2078 C369 P50139 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 536927 T2078 C369 P50139 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 536927 T2078 C369 P50139 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4] 2> 536928 T2120 C367 P56425 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 536928 T2120 C367 P56425 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 536930 T2074 C369 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 536931 T2120 C367 P56425 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4] 2> 536931 T2120 C367 P56425 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 536932 T2120 C367 P56425 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 536934 T2077 C369 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=3&version=2} status=0 QTime=0
[junit4:junit4] 2> 536935 T2120 C367 P56425 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 536936 T2120 C367 P56425 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index.20130618012843953
[junit4:junit4] 2> 536936 T2120 C367 P56425 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index.20130618012843953 lockFactory=org.apache.lucene.store.NativeFSLockFactory@56e86714 fullCopy=false
[junit4:junit4] 2> 536939 T2076 C369 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/replication params={file=segments_3&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=3} status=0 QTime=0
[junit4:junit4] 2> ASYNC NEW_CORE C370 name=collection1 org.apache.solr.core.SolrCore@f65639e url=http://127.0.0.1:56425/_wvf/collection1 node=127.0.0.1:56425__wvf C370_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:56425__wvf, base_url=http://127.0.0.1:56425/_wvf}
[junit4:junit4] 2> 536947 T2120 C370 P56425 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 536950 T2120 C370 P56425 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 536951 T2120 C370 P56425 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 536954 T2120 C370 P56425 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> 536955 T2120 C370 P56425 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 536955 T2120 C370 P56425 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 536956 T2120 C370 P56425 oass.SolrIndexSearcher.<init> Opening Searcher@162718e5 main
[junit4:junit4] 2> 536957 T2119 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@162718e5 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 536957 T2120 C370 P56425 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index.20130618012843953 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index.20130618012843953;done=true>>]
[junit4:junit4] 2> 536958 T2120 C370 P56425 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index.20130618012843953
[junit4:junit4] 2> 536958 T2120 C370 P56425 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index.20130618012843953
[junit4:junit4] 2> 536958 T2120 C370 P56425 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 536958 T2120 C370 P56425 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 536959 T2120 C370 P56425 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 536959 T2120 C370 P56425 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 536960 T2120 C370 P56425 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 537078 T2137 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371500920926/collection1
[junit4:junit4] 2> 537079 T2137 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 537079 T2137 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 537080 T2137 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 537081 T2137 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371500920926/collection1/'
[junit4:junit4] 2> 537081 T2137 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty4-1371500920926/collection1/lib/README' to classloader
[junit4:junit4] 2> 537082 T2137 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty4-1371500920926/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 537118 T2137 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 537154 T2137 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 537156 T2137 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 537159 T2137 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 537516 T2137 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 537523 T2137 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 537526 T2137 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 537539 T2137 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 537542 T2137 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 537545 T2137 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 537547 T2137 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 537547 T2137 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 537547 T2137 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 537548 T2137 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 537548 T2137 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 537549 T2137 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 537549 T2137 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371500920926/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/
[junit4:junit4] 2> 537549 T2137 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@39061e68
[junit4:junit4] 2> 537549 T2137 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 537550 T2137 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4
[junit4:junit4] 2> 537550 T2137 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index/
[junit4:junit4] 2> 537550 T2137 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 537551 T2137 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index
[junit4:junit4] 2> 537568 T2137 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 537568 T2137 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 537570 T2137 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 537571 T2137 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 537572 T2137 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 537572 T2137 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 537573 T2137 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 537573 T2137 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 537574 T2137 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 537574 T2137 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 537575 T2137 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 537577 T2137 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 537580 T2137 oass.SolrIndexSearcher.<init> Opening Searcher@6edd6f5c main
[junit4:junit4] 2> 537581 T2137 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 537581 T2137 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 537586 T2140 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6edd6f5c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 537587 T2137 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 537588 T2137 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54789/_wvf collection:collection1 shard:shard1
[junit4:junit4] 2> 537591 T2137 oasc.ZkController.register We are http://127.0.0.1:54789/_wvf/collection1/ and leader is http://127.0.0.1:50139/_wvf/collection1/
[junit4:junit4] 2> 537591 T2137 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54789/_wvf
[junit4:junit4] 2> 537591 T2137 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 537592 T2137 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C371 name=collection1 org.apache.solr.core.SolrCore@3e9d8e5 url=http://127.0.0.1:54789/_wvf/collection1 node=127.0.0.1:54789__wvf C371_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:54789__wvf, base_url=http://127.0.0.1:54789/_wvf}
[junit4:junit4] 2> 537592 T2141 C371 P54789 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 537593 T2141 C371 P54789 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 537593 T2141 C371 P54789 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 537593 T2137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 537593 T2141 C371 P54789 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 537595 T2040 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 537595 T2040 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 537596 T2040 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 537599 T2141 C371 P54789 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 537600 T2040 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 537601 T2040 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 537601 T2040 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 537602 T2040 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 537602 T2074 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:54789__wvf_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 537617 T2063 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 537618 T2063 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:56425__wvf_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56425__wvf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56425/_wvf"}
[junit4:junit4] 2> 537621 T2063 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:54789__wvf_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:54789__wvf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54789/_wvf"}
[junit4:junit4] 2> 537626 T2084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 537626 T2100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 537626 T2117 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 537626 T2069 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 537626 T2134 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 537626 T2062 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 538602 T2074 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:54789__wvf_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 538603 T2074 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:54789__wvf_collection1&state=recovering&nodeName=127.0.0.1:54789__wvf&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 538603 T2040 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 539605 T2040 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C371_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:54789__wvf, base_url=http://127.0.0.1:54789/_wvf}
[junit4:junit4] 2> 540604 T2141 C371 P54789 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:50139/_wvf/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 540604 T2141 C371 P54789 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:54789/_wvf START replicas=[http://127.0.0.1:50139/_wvf/collection1/] nUpdates=100
[junit4:junit4] 2> 540604 T2141 C371 P54789 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 540605 T2141 C371 P54789 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 540605 T2141 C371 P54789 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 540605 T2141 C371 P54789 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 540605 T2141 C371 P54789 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 540605 T2141 C371 P54789 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:50139/_wvf/collection1/. core=collection1
[junit4:junit4] 2> 540605 T2141 C371 P54789 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 540606 T2040 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C372 name=collection1 org.apache.solr.core.SolrCore@1dd44625 url=http://127.0.0.1:50139/_wvf/collection1 node=127.0.0.1:50139__wvf C372_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:50139__wvf, base_url=http://127.0.0.1:50139/_wvf, leader=true}
[junit4:junit4] 2> 540607 T2076 C372 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 540609 T2078 C372 P50139 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 540616 T2078 C372 P50139 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> 540617 T2078 C372 P50139 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 540618 T2078 C372 P50139 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 540618 T2078 C372 P50139 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4] 2> 540619 T2141 C371 P54789 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 540619 T2141 C371 P54789 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 540621 T2077 C372 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 540621 T2141 C371 P54789 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4] 2> 540622 T2141 C371 P54789 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 540622 T2141 C371 P54789 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 540624 T2076 C372 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=4&version=2} status=0 QTime=1
[junit4:junit4] 2> 540624 T2141 C371 P54789 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 540625 T2141 C371 P54789 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index.20130618012847642
[junit4:junit4] 2> 540625 T2141 C371 P54789 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index.20130618012847642 lockFactory=org.apache.lucene.store.NativeFSLockFactory@517a78a4 fullCopy=false
[junit4:junit4] 2> 540628 T2074 C372 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/replication params={file=segments_4&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=4} status=0 QTime=0
[junit4:junit4] 2> 540633 T2141 C371 P54789 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 540635 T2141 C371 P54789 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 540636 T2141 C371 P54789 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 540638 T2141 C371 P54789 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> 540638 T2141 C371 P54789 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 540639 T2141 C371 P54789 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 540639 T2141 C371 P54789 oass.SolrIndexSearcher.<init> Opening Searcher@3cf11d24 main
[junit4:junit4] 2> 540640 T2140 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3cf11d24 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 540640 T2141 C371 P54789 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index.20130618012847642 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index.20130618012847642;done=true>>]
[junit4:junit4] 2> 540641 T2141 C371 P54789 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index.20130618012847642
[junit4:junit4] 2> 540641 T2141 C371 P54789 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index.20130618012847642
[junit4:junit4] 2> 540641 T2141 C371 P54789 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 540641 T2141 C371 P54789 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 540642 T2141 C371 P54789 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 540642 T2141 C371 P54789 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 540643 T2141 C371 P54789 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 541609 T2040 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 542134 T2063 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 542135 T2063 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:54789__wvf_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:54789__wvf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54789/_wvf"}
[junit4:junit4] 2> 542156 T2134 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 542157 T2069 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 542156 T2100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 542157 T2062 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 542157 T2084 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 542157 T2117 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 542610 T2040 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 542611 T2040 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C373 name=collection1 org.apache.solr.core.SolrCore@1f753ca5 url=http://127.0.0.1:36298/_wvf/collection1 node=127.0.0.1:36298__wvf C373_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:36298__wvf, base_url=http://127.0.0.1:36298/_wvf, leader=true}
[junit4:junit4] 2> 542617 T2057 C373 P36298 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 542620 T2057 C373 P36298 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 542621 T2057 C373 P36298 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 542626 T2057 C373 P36298 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/control/data/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 542627 T2057 C373 P36298 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 542628 T2057 C373 P36298 oass.SolrIndexSearcher.<init> Opening Searcher@78a073d5 main
[junit4:junit4] 2> 542628 T2057 C373 P36298 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 542629 T2066 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@78a073d5 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 542629 T2057 C373 P36298 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 12
[junit4:junit4] 2> C371_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:54789__wvf, base_url=http://127.0.0.1:54789/_wvf}
[junit4:junit4] 2> 542637 T2128 C371 P54789 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:50139/_wvf/collection1/, StdNode: http://127.0.0.1:39311/_wvf/collection1/, StdNode: http://127.0.0.1:56425/_wvf/collection1/, StdNode: http://127.0.0.1:54789/_wvf/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 542641 T2077 C372 P50139 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 542643 T2130 C371 P54789 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C374 name=collection1 org.apache.solr.core.SolrCore@497bb53c url=http://127.0.0.1:39311/_wvf/collection1 node=127.0.0.1:39311__wvf C374_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:39311__wvf, base_url=http://127.0.0.1:39311/_wvf}
[junit4:junit4] 2> 542646 T2095 C374 P39311 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C375 name=collection1 org.apache.solr.core.SolrCore@f65639e url=http://127.0.0.1:56425/_wvf/collection1 node=127.0.0.1:56425__wvf C375_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:56425__wvf, base_url=http://127.0.0.1:56425/_wvf}
[junit4:junit4] 2> 542646 T2112 C375 P56425 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 542656 T2077 C372 P50139 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=4
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> 542657 T2077 C372 P50139 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 542658 T2077 C372 P50139 oass.SolrIndexSearcher.<init> Opening Searcher@44ac55ae main
[junit4:junit4] 2> 542658 T2077 C372 P50139 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 542659 T2086 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@44ac55ae main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 542659 T2077 C372 P50139 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 18
[junit4:junit4] 2> 542666 T2130 C371 P54789 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> 542667 T2130 C371 P54789 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 542668 T2130 C371 P54789 oass.SolrIndexSearcher.<init> Opening Searcher@32eb53df main
[junit4:junit4] 2> 542668 T2130 C371 P54789 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 542669 T2140 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@32eb53df main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 542669 T2130 C371 P54789 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 26
[junit4:junit4] 2> 542686 T2095 C374 P39311 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> 542687 T2095 C374 P39311 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 542689 T2095 C374 P39311 oass.SolrIndexSearcher.<init> Opening Searcher@b67c23 main
[junit4:junit4] 2> 542690 T2095 C374 P39311 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 542691 T2102 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b67c23 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 542692 T2095 C374 P39311 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 46
[junit4:junit4] 2> 542696 T2112 C375 P56425 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> 542697 T2112 C375 P56425 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 542698 T2112 C375 P56425 oass.SolrIndexSearcher.<init> Opening Searcher@13a5a0a1 main
[junit4:junit4] 2> 542699 T2112 C375 P56425 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 542700 T2119 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13a5a0a1 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 542701 T2112 C375 P56425 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 55
[junit4:junit4] 2> 542702 T2128 C371 P54789 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 65
[junit4:junit4] 2> 542703 T2040 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 542706 T2076 C372 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 542707 T2091 C374 P39311 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 542709 T2111 C375 P56425 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 542711 T2127 C371 P54789 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 544715 T2057 C373 P36298 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1438122960991813632)} 0 1
[junit4:junit4] 2> 544726 T2111 C375 P56425 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={update.distrib=FROMLEADER&_version_=-1438122960998105088&update.from=http://127.0.0.1:50139/_wvf/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438122960998105088)} 0 1
[junit4:junit4] 2> 544726 T2096 C374 P39311 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={update.distrib=FROMLEADER&_version_=-1438122960998105088&update.from=http://127.0.0.1:50139/_wvf/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438122960998105088)} 0 2
[junit4:junit4] 2> 544727 T2130 C371 P54789 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={update.distrib=FROMLEADER&_version_=-1438122960998105088&update.from=http://127.0.0.1:50139/_wvf/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438122960998105088)} 0 1
[junit4:junit4] 2> 544728 T2078 C372 P50139 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1438122960998105088)} 0 8
[junit4:junit4] 2> 544728 T2128 C371 P54789 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 10
[junit4:junit4] 2> 544734 T2058 C373 P36298 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={wt=javabin&version=2} {add=[0 (1438122961009639424)]} 0 3
[junit4:junit4] 2> 544745 T2113 C375 P56425 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={distrib.from=http://127.0.0.1:50139/_wvf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438122961019076608)]} 0 1
[junit4:junit4] 2> 544745 T2127 C371 P54789 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={distrib.from=http://127.0.0.1:50139/_wvf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438122961019076608)]} 0 2
[junit4:junit4] 2> 544748 T2095 C374 P39311 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={distrib.from=http://127.0.0.1:50139/_wvf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438122961019076608)]} 0 2
[junit4:junit4] 2> 544749 T2077 C372 P50139 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={distrib.from=http://127.0.0.1:39311/_wvf/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1438122961019076608)]} 0 9
[junit4:junit4] 2> 544749 T2094 C374 P39311 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={wt=javabin&version=2} {add=[0]} 0 12
[junit4:junit4] 2> 544752 T2055 C373 P36298 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={wt=javabin&version=2} {add=[1 (1438122961031659520)]} 0 1
[junit4:junit4] 2> 544761 T2112 C375 P56425 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={distrib.from=http://127.0.0.1:50139/_wvf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438122961036902400)]} 0 1
[junit4:junit4] 2> 544762 T2091 C374 P39311 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={distrib.from=http://127.0.0.1:50139/_wvf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438122961036902400)]} 0 2
[junit4:junit4] 2> 544762 T2126 C371 P54789 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={distrib.from=http://127.0.0.1:50139/_wvf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438122961036902400)]} 0 2
[junit4:junit4] 2> 544763 T2076 C372 P50139 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={distrib.from=http://127.0.0.1:39311/_wvf/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1438122961036902400)]} 0 6
[junit4:junit4] 2> 544764 T2096 C374 P39311 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={wt=javabin&version=2} {add=[1]} 0 9
[junit4:junit4] 2> 544768 T2056 C373 P36298 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={wt=javabin&version=2} {add=[2 (1438122961047388160)]} 0 1
[junit4:junit4] 2> 544772 T2074 C372 P50139 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:39311/_wvf/collection1/]
[junit4:junit4] 2> 544773 T2074 C372 P50139 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:39311/_wvf/collection1/ against:[http://127.0.0.1:39311/_wvf/collection1/] result:true
[junit4:junit4] 2> 544773 T2074 C372 P50139 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:56425/_wvf/collection1/ against:[http://127.0.0.1:39311/_wvf/collection1/] result:false
[junit4:junit4] 2> 544774 T2074 C372 P50139 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:54789/_wvf/collection1/ against:[http://127.0.0.1:39311/_wvf/collection1/] result:false
[junit4:junit4] 2> 544778 T2107 C375 P56425 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={distrib.from=http://127.0.0.1:50139/_wvf/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:39311/_wvf/collection1/&wt=javabin&version=2} {add=[2 (1438122961054728192)]} 0 1
[junit4:junit4] 2> 544779 T2128 C371 P54789 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={distrib.from=http://127.0.0.1:50139/_wvf/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:39311/_wvf/collection1/&wt=javabin&version=2} {add=[2 (1438122961054728192)]} 0 0
[junit4:junit4] 2> 544780 T2074 C372 P50139 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={distrib.from=http://127.0.0.1:54789/_wvf/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:39311/_wvf/collection1/&wt=javabin&version=2} {add=[2 (1438122961054728192)]} 0 8
[junit4:junit4] 2> 544781 T2130 C371 P54789 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={test.distrib.skip.servers=http://127.0.0.1:39311/_wvf/collection1/&wt=javabin&version=2} {add=[2]} 0 11
[junit4:junit4] 2> 544784 T2057 C373 P36298 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={wt=javabin&version=2} {add=[3 (1438122961064165376)]} 0 1
[junit4:junit4] 2> 544789 T2079 C372 P50139 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:39311/_wvf/collection1/]
[junit4:junit4] 2> 544789 T2079 C372 P50139 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:39311/_wvf/collection1/ against:[http://127.0.0.1:39311/_wvf/collection1/] result:true
[junit4:junit4] 2> 544790 T2079 C372 P50139 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:56425/_wvf/collection1/ against:[http://127.0.0.1:39311/_wvf/collection1/] result:false
[junit4:junit4] 2> 544790 T2079 C372 P50139 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:54789/_wvf/collection1/ against:[http://127.0.0.1:39311/_wvf/collection1/] result:false
[junit4:junit4] 2> 544794 T2126 C371 P54789 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={distrib.from=http://127.0.0.1:50139/_wvf/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:39311/_wvf/collection1/&wt=javabin&version=2} {add=[3 (1438122961071505408)]} 0 1
[junit4:junit4] 2> 544794 T2111 C375 P56425 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={distrib.from=http://127.0.0.1:50139/_wvf/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:39311/_wvf/collection1/&wt=javabin&version=2} {add=[3 (1438122961071505408)]} 0 1
[junit4:junit4] 2> 544795 T2079 C372 P50139 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={distrib.from=http://127.0.0.1:54789/_wvf/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:39311/_wvf/collection1/&wt=javabin&version=2} {add=[3 (1438122961071505408)]} 0 6
[junit4:junit4] 2> 544796 T2127 C371 P54789 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={test.distrib.skip.servers=http://127.0.0.1:39311/_wvf/collection1/&test.distrib.skip.servers=http://127.0.0.1:56425/_wvf/collection1/&wt=javabin&version=2} {add=[3]} 0 10
[junit4:junit4] 2> 544798 T2058 C373 P36298 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 544836 T2058 C373 P36298 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/control/data/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/control/data/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> 544837 T2058 C373 P36298 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 544840 T2058 C373 P36298 oass.SolrIndexSearcher.<init> Opening Searcher@28efd58 main
[junit4:junit4] 2> 544840 T2058 C373 P36298 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 544841 T2066 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@28efd58 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 544842 T2058 C373 P36298 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 44
[junit4:junit4] 2> 544844 T2078 C372 P50139 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:50139/_wvf/collection1/, StdNode: http://127.0.0.1:39311/_wvf/collection1/, StdNode: http://127.0.0.1:56425/_wvf/collection1/, StdNode: http://127.0.0.1:54789/_wvf/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 544846 T2076 C372 P50139 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 544846 T2128 C371 P54789 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 544846 T2095 C374 P39311 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 544847 T2113 C375 P56425 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 544886 T2113 C375 P56425 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> 544887 T2113 C375 P56425 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 544889 T2113 C375 P56425 oass.SolrIndexSearcher.<init> Opening Searcher@3af23c62 main
[junit4:junit4] 2> 544890 T2113 C375 P56425 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 544890 T2119 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3af23c62 main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 544891 T2113 C375 P56425 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 44
[junit4:junit4] 2> 544907 T2076 C372 P50139 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_6,generation=6}
[junit4:junit4] 2> 544908 T2076 C372 P50139 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 544911 T2076 C372 P50139 oass.SolrIndexSearcher.<init> Opening Searcher@7a985e83 main
[junit4:junit4] 2> 544911 T2076 C372 P50139 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 544912 T2086 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7a985e83 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 544912 T2076 C372 P50139 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 66
[junit4:junit4] 2> 544916 T2128 C371 P54789 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index,segFN=segments_6,generation=6}
[junit4:junit4] 2> 544917 T2128 C371 P54789 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 544919 T2128 C371 P54789 oass.SolrIndexSearcher.<init> Opening Searcher@4c192e0b main
[junit4:junit4] 2> 544920 T2128 C371 P54789 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 544924 T2140 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c192e0b main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 544924 T2128 C371 P54789 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 78
[junit4:junit4] 2> 544946 T2095 C374 P39311 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> 544947 T2095 C374 P39311 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 544949 T2095 C374 P39311 oass.SolrIndexSearcher.<init> Opening Searcher@2bd591a main
[junit4:junit4] 2> 544950 T2095 C374 P39311 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 544950 T2102 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2bd591a main{StandardDirectoryReader(segments_4:3:nrt _0(5.0):c2)}
[junit4:junit4] 2> 544951 T2095 C374 P39311 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 105
[junit4:junit4] 2> 544952 T2078 C372 P50139 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 108
[junit4:junit4] 2> 544953 T2040 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 544954 T2040 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 544954 T2040 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 544957 T2074 C372 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 544959 T2094 C374 P39311 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0
[junit4:junit4] 2> 544961 T2112 C375 P56425 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 544964 T2130 C371 P54789 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 544966 T2040 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 544972 T2111 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4] 2> 544973 T2111 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 544976 T2077 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4] 2> 544977 T2077 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50139/_wvf/collection1/
[junit4:junit4] 2> 544977 T2077 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50139/_wvf START replicas=[http://127.0.0.1:39311/_wvf/collection1/, http://127.0.0.1:56425/_wvf/collection1/, http://127.0.0.1:54789/_wvf/collection1/] nUpdates=100
[junit4:junit4] 2> 544979 T2091 C374 P39311 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 544979 T2077 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:50139/_wvf Received 3 versions from 127.0.0.1:39311/_wvf/collection1/
[junit4:junit4] 2> 544980 T2077 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:50139/_wvf Our versions are newer. ourLowThreshold=1438122960998105088 otherHigh=1438122961036902400
[junit4:junit4] 2> 544980 T2127 C371 P54789 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 544981 T2077 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:50139/_wvf Received 5 versions from 127.0.0.1:54789/_wvf/collection1/
[junit4:junit4] 2> 544981 T2077 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:50139/_wvf Our versions are newer. ourLowThreshold=1438122960998105088 otherHigh=1438122961054728192
[junit4:junit4] 2> 544983 T2113 C375 P56425 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 544984 T2077 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:50139/_wvf Received 5 versions from 127.0.0.1:56425/_wvf/collection1/
[junit4:junit4] 2> 544984 T2077 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:50139/_wvf Our versions are newer. ourLowThreshold=1438122960998105088 otherHigh=1438122961054728192
[junit4:junit4] 2> 544985 T2077 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50139/_wvf DONE. sync succeeded
[junit4:junit4] 2> 544985 T2077 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 544985 T2077 oasc.SyncStrategy.syncToMe http://127.0.0.1:50139/_wvf/collection1/: try and ask http://127.0.0.1:39311/_wvf/collection1/ to sync
[junit4:junit4] 2> 544986 T2077 oasc.SyncStrategy.syncToMe http://127.0.0.1:50139/_wvf/collection1/: try and ask http://127.0.0.1:56425/_wvf/collection1/ to sync
[junit4:junit4] 2> 544986 T2077 oasc.SyncStrategy.syncToMe http://127.0.0.1:50139/_wvf/collection1/: try and ask http://127.0.0.1:54789/_wvf/collection1/ to sync
[junit4:junit4] 2> 544987 T2096 C374 P39311 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39311/_wvf START replicas=[http://127.0.0.1:50139/_wvf/collection1/] nUpdates=100
[junit4:junit4] 2> 544987 T2107 C375 P56425 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56425/_wvf START replicas=[http://127.0.0.1:50139/_wvf/collection1/] nUpdates=100
[junit4:junit4] 2> 544987 T2128 C371 P54789 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:54789/_wvf START replicas=[http://127.0.0.1:50139/_wvf/collection1/] nUpdates=100
[junit4:junit4] 2> 544989 T2079 C372 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 544989 T2076 C372 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 544990 T2128 C371 P54789 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:54789/_wvf Received 5 versions from 127.0.0.1:50139/_wvf/collection1/
[junit4:junit4] 2> 544989 T2074 C372 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 544991 T2128 C371 P54789 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:54789/_wvf Our versions are newer. ourLowThreshold=1438122960998105088 otherHigh=1438122961054728192
[junit4:junit4] 2> 544990 T2096 C374 P39311 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39311/_wvf Received 5 versions from 127.0.0.1:50139/_wvf/collection1/
[junit4:junit4] 2> 544992 T2128 C371 P54789 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:54789/_wvf DONE. sync succeeded
[junit4:junit4] 2> 544992 T2128 C371 P54789 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/get params={sync=http://127.0.0.1:50139/_wvf/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=5
[junit4:junit4] 2> 544991 T2107 C375 P56425 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:56425/_wvf Received 5 versions from 127.0.0.1:50139/_wvf/collection1/
[junit4:junit4] 2> 544993 T2077 oasc.SyncStrategy.syncToMe http://127.0.0.1:50139/_wvf/collection1/: sync completed with http://127.0.0.1:54789/_wvf/collection1/
[junit4:junit4] 2> 544992 T2096 C374 P39311 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:39311/_wvf Requesting updates from 127.0.0.1:50139/_wvf/collection1/n=2 versions=[1438122961071505408, 1438122961054728192]
[junit4:junit4] 2> 544993 T2107 C375 P56425 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:56425/_wvf Our versions are newer. ourLowThreshold=1438122960998105088 otherHigh=1438122961054728192
[junit4:junit4] 2> 544994 T2107 C375 P56425 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56425/_wvf DONE. sync succeeded
[junit4:junit4] 2> 544994 T2107 C375 P56425 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/get params={sync=http://127.0.0.1:50139/_wvf/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=7
[junit4:junit4] 2> 544995 T2077 oasc.SyncStrategy.syncToMe http://127.0.0.1:50139/_wvf/collection1/: sync completed with http://127.0.0.1:56425/_wvf/collection1/
[junit4:junit4] 2> 544995 T2078 C372 P50139 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/get params={distrib=false&getUpdates=1438122961071505408,1438122961054728192&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 544999 T2096 C374 P39311 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1438122961054728192), 3 (1438122961071505408)]} 0 3
[junit4:junit4] 2> 544999 T2096 C374 P39311 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39311/_wvf DONE. sync succeeded
[junit4:junit4] 2> 545000 T2096 C374 P39311 oasc.SolrCore.execute [collection1] webapp=/_wvf path=/get params={sync=http://127.0.0.1:50139/_wvf/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=13
[junit4:junit4] 2> 545000 T2077 oasc.SyncStrategy.syncToMe http://127.0.0.1:50139/_wvf/collection1/: sync completed with http://127.0.0.1:39311/_wvf/collection1/
[junit4:junit4] 2> 545001 T2077 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=25
[junit4:junit4] 2> 545001 T2111 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=30
[junit4:junit4] 2> 545001 T2040 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 545002 T2040 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 545002 T2040 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 545003 T2040 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 545005 T2056 C373 P36298 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 545036 T2056 C373 P36298 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/control/data/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/control/data/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> 545037 T2056 C373 P36298 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 545038 T2056 C373 P36298 oass.SolrIndexSearcher.<init> Opening Searcher@73e01888 main
[junit4:junit4] 2> 545038 T2056 C373 P36298 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 545039 T2066 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@73e01888 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 545040 T2056 C373 P36298 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 35
[junit4:junit4] 2> 545042 T2079 C372 P50139 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:50139/_wvf/collection1/, StdNode: http://127.0.0.1:39311/_wvf/collection1/, StdNode: http://127.0.0.1:56425/_wvf/collection1/, StdNode: http://127.0.0.1:54789/_wvf/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 545044 T2130 C371 P54789 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 545045 T2076 C372 P50139 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 545044 T2112 C375 P56425 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 545044 T2095 C374 P39311 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 545066 T2130 C371 P54789 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index,segFN=segments_6,generation=6}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty4/index,segFN=segments_7,generation=7}
[junit4:junit4] 2> 545067 T2130 C371 P54789 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 7
[junit4:junit4] 2> 545068 T2130 C371 P54789 oass.SolrIndexSearcher.<init> Opening Searcher@3687aae0 main
[junit4:junit4] 2> 545068 T2130 C371 P54789 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 545069 T2140 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3687aae0 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 545069 T2130 C371 P54789 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 25
[junit4:junit4] 2> 545086 T2112 C375 P56425 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty3/index,segFN=segments_6,generation=6}
[junit4:junit4] 2> 545087 T2112 C375 P56425 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 545088 T2112 C375 P56425 oass.SolrIndexSearcher.<init> Opening Searcher@70dad629 main
[junit4:junit4] 2> 545088 T2112 C375 P56425 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 545089 T2076 C372 P50139 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=4
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_6,generation=6}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty1/index,segFN=segments_7,generation=7}
[junit4:junit4] 2> 545089 T2119 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@70dad629 main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 545089 T2076 C372 P50139 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 7
[junit4:junit4] 2> 545090 T2076 C372 P50139 oass.SolrIndexSearcher.<init> Opening Searcher@77a72333 main
[junit4:junit4] 2> 545091 T2076 C372 P50139 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 545091 T2112 C375 P56425 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 47
[junit4:junit4] 2> 545091 T2086 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@77a72333 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 545092 T2076 C372 P50139 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 47
[junit4:junit4] 2> 545126 T2095 C374 P39311 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-1371500907603/jetty2/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> 545127 T2095 C374 P39311 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 545129 T2095 C374 P39311 oass.SolrIndexSearcher.<init> Opening Searcher@7a34fbbb main
[junit4:junit4] 2> 545130 T2095 C374 P39311 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 545131 T2102 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7a34fbbb main{StandardDirectoryReader(segments_5:5:nrt _0(5.0):c2 _1(5.0):c2)}
[junit4:junit4] 2> 545131 T2095 C374 P39311 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 87
[junit4:junit4] 2> 545132 T2079 C372 P50139 oasup.LogUpdateProcessor.finish [collection1] webapp=/_wvf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 90
[junit4:junit4] 2> 545133 T2040 oascc.ZkState
[...truncated too long message...]
icas to http://127.0.0.1:43251/collection1/
[junit4:junit4] 2> 804863 T2616 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 804863 T2616 oasc.SyncStrategy.syncToMe http://127.0.0.1:43251/collection1/ has no replicas
[junit4:junit4] 2> 804863 T2616 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43251/collection1/
[junit4:junit4] 2> 804863 T2616 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 804864 T2650 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:40977__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40977_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40977"}
[junit4:junit4] 2> 804878 T2532 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 804882 T2616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 804930 T2532 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 43251
[junit4:junit4] 2> 804931 T2532 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1197223709
[junit4:junit4] 2> 805932 T2532 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 805933 T2532 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 805934 T2532 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@434efd00
[junit4:junit4] 2> 805941 T2532 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=43,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=46,cumulative_deletesById=16,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 805942 T2532 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 805943 T2532 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 805943 T2532 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 805944 T2532 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 805945 T2532 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 805946 T2532 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371501058069/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371501058069/jetty2;done=false>>]
[junit4:junit4] 2> 805946 T2532 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371501058069/jetty2
[junit4:junit4] 2> 805946 T2532 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1371501058069/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1371501058069/jetty2/index;done=false>>]
[junit4:junit4] 2> 805947 T2532 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1371501058069/jetty2/index
[junit4:junit4] 2> 805948 T2650 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89882693341806602-127.0.0.1:43251_-n_0000000003) am no longer a leader.
[junit4:junit4] 2> 805972 T2532 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 806037 T2532 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 806039 T2532 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:56002 56002
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.method=testDistribSearch -Dtests.seed=132BBD2C8A8DA51 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=lv_LV -Dtests.timezone=America/Porto_Velho -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE 135s J1 | RecoveryZkTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: expected:<588> but was:<32>
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([132BBD2C8A8DA51:80D435CABFF7BA6D]:0)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:108)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 806055 T2532 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 135011 T2531 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 807250 T2616 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 807250 T2616 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 807250 T2616 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=Lucene41(blocksize=128), text=PostingsFormat(name=Lucene41WithOrds), _version_=PostingsFormat(name=Direct), rnd_b=PostingsFormat(name=Memory doPackFST= true), intDefault=PostingsFormat(name=Direct), timestamp=Lucene41(blocksize=128), id=PostingsFormat(name=Memory doPackFST= true), a_t=Lucene41(blocksize=128), range_facet_sl=PostingsFormat(name=Memory doPackFST= true), range_facet_si=Lucene41(blocksize=128), other_tl1=PostingsFormat(name=Direct), multiDefault=Lucene41(blocksize=128), a_si=PostingsFormat(name=Direct)}, docValues:{timestamp=DocValuesFormat(name=SimpleText)}, sim=DefaultSimilarity, locale=lv_LV, timezone=America/Porto_Velho
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=8,threads=1,free=141450120,total=380043264
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestDefaultSearchFieldResource, IndexReaderFactoryTest, TestJmxIntegration, TestDefaultSimilarityFactory, TestFastOutputStream, SolrRequestParserTest, TestBinaryResponseWriter, LeaderElectionIntegrationTest, IndexSchemaRuntimeFieldTest, TestCollationField, SpellCheckComponentTest, SpellCheckCollatorTest, TestElisionMultitermQuery, TestSolrXml, TestUtils, TestSolrJ, TestAnalyzedSuggestions, OpenExchangeRatesOrgProviderTest, TestDynamicFieldCollectionResource, TestReload, TestPartialUpdateDeduplication, SolrCmdDistributorTest, TestSolrXMLSerializer, TestComponentsName, PingRequestHandlerTest, SignatureUpdateProcessorFactoryTest, TestSuggestSpellingConverter, TestArbitraryIndexDir, TimeZoneUtilsTest, FieldAnalysisRequestHandlerTest, TestRecovery, StatsComponentTest, ClusterStateTest, TestSolr4Spatial, TestHashPartitioner, TestNumberUtils, CopyFieldTest, TestShardHandlerFactory, ConvertedLegacyTest, TestDFRSimilarityFactory, TestCopyFieldCollectionResource, BasicDistributedZk2Test, TestCSVResponseWriter, NoCacheHeaderTest, TestOmitPositions, ChaosMonkeySafeLeaderTest, HighlighterTest, SimplePostToolTest, CoreContainerCoreInitFailuresTest, StandardRequestHandlerTest, TestDistributedGrouping, TestMultiCoreConfBootstrap, NotRequiredUniqueKeyTest, NumericFieldsTest, SolrIndexConfigTest, TestCodecSupport, TermVectorComponentDistributedTest, RequestHandlersTest, TestDynamicFieldResource, HighlighterConfigTest, SortByFunctionTest, TestDistributedSearch, ZkNodePropsTest, TestSolrQueryParserDefaultOperatorResource, TestAddFieldRealTimeGet, FileUtilsTest, SampleTest, HardAutoCommitTest, TestSolrQueryParserResource, XmlUpdateRequestHandlerTest, TestManagedSchemaFieldResource, UnloadDistributedZkTest, SyncSliceTest, ShardRoutingTest, RecoveryZkTest]
[junit4:junit4] Completed on J1 in 136.29s, 1 test, 1 failure <<< FAILURES!
[...truncated 432 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1249 tests, 2 failures, 13 ignored (7 assumptions)
Total time: 36 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0-ea-b93) - Build #
6180 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6180/
Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseSerialGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 634 from http://127.0.0.1:45848/ge/collection1lastClient and got 689 from http://127.0.0.1:55405/ge/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 634 from http://127.0.0.1:45848/ge/collection1lastClient and got 689 from http://127.0.0.1:55405/ge/collection1
at __randomizedtesting.SeedInfo.seed([472419F524A213F6:C6C297ED53FD73CA]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:491)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9315 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 19658 T24 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ge/
[junit4:junit4] 2> 19674 T24 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1371495604517
[junit4:junit4] 2> 19676 T24 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 19679 T25 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 19778 T24 oasc.ZkTestServer.run start zk server on port:32827
[junit4:junit4] 2> 19819 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 19957 T31 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1749998 name:ZooKeeperConnection Watcher:127.0.0.1:32827 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 19957 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 19959 T24 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 19983 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 19986 T33 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bcb925 name:ZooKeeperConnection Watcher:127.0.0.1:32827/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 19986 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 19990 T24 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 19995 T24 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 19999 T24 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 20001 T24 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 20006 T24 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 20013 T24 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 20020 T24 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 20020 T24 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 20025 T24 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 20025 T24 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 20029 T24 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 20029 T24 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 20033 T24 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 20033 T24 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 20037 T24 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 20037 T24 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 20041 T24 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 20042 T24 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 20046 T24 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 20046 T24 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 20050 T24 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 20051 T24 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 20313 T24 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 20405 T24 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42533
[junit4:junit4] 2> 20420 T24 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 20421 T24 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 20421 T24 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371495604899
[junit4:junit4] 2> 20422 T24 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371495604899/solr.xml
[junit4:junit4] 2> 20422 T24 oasc.CoreContainer.<init> New CoreContainer 28915695
[junit4:junit4] 2> 20423 T24 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371495604899/'
[junit4:junit4] 2> 20423 T24 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371495604899/'
[junit4:junit4] 2> 20517 T24 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 20518 T24 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 20518 T24 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 20519 T24 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 20519 T24 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 20519 T24 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 20520 T24 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 20520 T24 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 20520 T24 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 20532 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 20779 T24 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 20781 T24 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:32827/solr
[junit4:junit4] 2> 20791 T24 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 20792 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 20795 T46 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19122a name:ZooKeeperConnection Watcher:127.0.0.1:32827 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 20795 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 20799 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 20814 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 20816 T48 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9e04ff name:ZooKeeperConnection Watcher:127.0.0.1:32827/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 20817 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 20823 T24 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 20832 T24 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 20840 T24 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 20843 T24 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42533_ge
[junit4:junit4] 2> 20846 T24 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42533_ge
[junit4:junit4] 2> 20852 T24 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 20866 T24 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 20869 T24 oasc.Overseer.start Overseer (id=89882335942737923-127.0.0.1:42533_ge-n_0000000000) starting
[junit4:junit4] 2> 20880 T24 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 20892 T50 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 20892 T24 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 20897 T24 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 20903 T24 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 20919 T49 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 20943 T51 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 20943 T51 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 22431 T49 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 22432 T49 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42533/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:42533_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"control_collection"}
[junit4:junit4] 2> 22432 T49 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 22440 T49 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 22450 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 22946 T51 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371495604899/collection1
[junit4:junit4] 2> 22946 T51 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 22947 T51 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 22947 T51 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 22950 T51 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371495604899/collection1/'
[junit4:junit4] 2> 22951 T51 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371495604899/collection1/lib/README' to classloader
[junit4:junit4] 2> 22951 T51 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371495604899/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 23002 T51 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 23068 T51 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 23070 T51 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 23092 T51 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 23602 T51 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 23613 T51 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 23616 T51 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 23651 T51 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 23656 T51 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 23661 T51 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 23663 T51 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 23663 T51 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 23664 T51 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 23665 T51 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 23666 T51 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 23666 T51 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 23677 T51 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371495604899/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/control/data/
[junit4:junit4] 2> 23678 T51 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1b67c33
[junit4:junit4] 2> 23679 T51 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 23680 T51 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/control/data
[junit4:junit4] 2> 23680 T51 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/control/data/index/
[junit4:junit4] 2> 23680 T51 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 23681 T51 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/control/data/index
[junit4:junit4] 2> 23685 T51 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 23685 T51 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 23706 T51 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 23707 T51 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 23708 T51 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 23709 T51 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 23710 T51 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 23710 T51 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 23711 T51 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 23711 T51 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 23712 T51 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 23715 T51 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 23719 T51 oass.SolrIndexSearcher.<init> Opening Searcher@1c47176 main
[junit4:junit4] 2> 23719 T51 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 23720 T51 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 23724 T52 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c47176 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 23727 T51 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 23727 T51 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42533/ge collection:control_collection shard:shard1
[junit4:junit4] 2> 23733 T51 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 23744 T51 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 23755 T51 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 23759 T51 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 23759 T51 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 23761 T51 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42533/ge/collection1/
[junit4:junit4] 2> 23762 T51 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 23762 T51 oasc.SyncStrategy.syncToMe http://127.0.0.1:42533/ge/collection1/ has no replicas
[junit4:junit4] 2> 23762 T51 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42533/ge/collection1/
[junit4:junit4] 2> 23763 T51 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 23955 T49 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 24000 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 24022 T51 oasc.ZkController.register We are http://127.0.0.1:42533/ge/collection1/ and leader is http://127.0.0.1:42533/ge/collection1/
[junit4:junit4] 2> 24022 T51 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42533/ge
[junit4:junit4] 2> 24023 T51 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 24023 T51 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 24023 T51 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 24025 T51 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 24027 T24 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 24028 T24 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 24047 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 24053 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 24055 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 24057 T55 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1512744 name:ZooKeeperConnection Watcher:127.0.0.1:32827/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 24058 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 24059 T24 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 24064 T24 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 24130 T24 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 24133 T24 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46820
[junit4:junit4] 2> 24134 T24 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 24134 T24 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 24134 T24 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371495608907
[junit4:junit4] 2> 24135 T24 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371495608907/solr.xml
[junit4:junit4] 2> 24135 T24 oasc.CoreContainer.<init> New CoreContainer 21580004
[junit4:junit4] 2> 24135 T24 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371495608907/'
[junit4:junit4] 2> 24136 T24 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371495608907/'
[junit4:junit4] 2> 24194 T24 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 24194 T24 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 24194 T24 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 24195 T24 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 24195 T24 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 24195 T24 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 24195 T24 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 24196 T24 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 24196 T24 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 24196 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 24200 T24 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 24201 T24 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:32827/solr
[junit4:junit4] 2> 24201 T24 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 24202 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 24204 T66 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f110f name:ZooKeeperConnection Watcher:127.0.0.1:32827 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 24204 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 24207 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 24210 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 24212 T68 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f48fc4 name:ZooKeeperConnection Watcher:127.0.0.1:32827/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 24212 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 24217 T24 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 25223 T24 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46820_ge
[junit4:junit4] 2> 25225 T24 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46820_ge
[junit4:junit4] 2> 25232 T55 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 25232 T48 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 25232 T68 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 25233 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 25243 T69 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 25243 T69 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 25509 T49 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 25512 T49 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42533/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:42533_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"control_collection"}
[junit4:junit4] 2> 25519 T49 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46820/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:46820_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 25519 T49 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 25519 T49 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 25524 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 25524 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 25524 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 26246 T69 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371495608907/collection1
[junit4:junit4] 2> 26246 T69 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 26247 T69 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 26247 T69 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 26249 T69 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371495608907/collection1/'
[junit4:junit4] 2> 26250 T69 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371495608907/collection1/lib/README' to classloader
[junit4:junit4] 2> 26250 T69 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371495608907/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 26283 T69 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 26323 T69 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 26325 T69 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 26335 T69 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 26740 T69 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 26750 T69 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 26755 T69 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 26784 T69 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 26788 T69 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 26791 T69 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 26793 T69 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 26793 T69 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 26793 T69 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 26795 T69 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 26795 T69 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 26795 T69 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 26795 T69 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371495608907/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty1/
[junit4:junit4] 2> 26796 T69 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1b67c33
[junit4:junit4] 2> 26797 T69 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 26798 T69 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty1
[junit4:junit4] 2> 26798 T69 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty1/index/
[junit4:junit4] 2> 26798 T69 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 26798 T69 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty1/index
[junit4:junit4] 2> 26801 T69 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 26802 T69 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 26804 T69 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 26805 T69 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 26805 T69 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 26806 T69 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 26807 T69 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 26808 T69 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 26808 T69 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 26809 T69 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 26810 T69 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 26812 T69 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 26817 T69 oass.SolrIndexSearcher.<init> Opening Searcher@91218f main
[junit4:junit4] 2> 26818 T69 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 26818 T69 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 26825 T70 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@91218f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 26829 T69 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 26829 T69 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46820/ge collection:collection1 shard:shard3
[junit4:junit4] 2> 26830 T69 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 26839 T69 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 26842 T69 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 26842 T69 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 26842 T69 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46820/ge/collection1/
[junit4:junit4] 2> 26842 T69 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 26843 T69 oasc.SyncStrategy.syncToMe http://127.0.0.1:46820/ge/collection1/ has no replicas
[junit4:junit4] 2> 26843 T69 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46820/ge/collection1/
[junit4:junit4] 2> 26843 T69 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 27031 T49 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 27040 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 27041 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 27040 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 27051 T69 oasc.ZkController.register We are http://127.0.0.1:46820/ge/collection1/ and leader is http://127.0.0.1:46820/ge/collection1/
[junit4:junit4] 2> 27051 T69 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46820/ge
[junit4:junit4] 2> 27051 T69 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 27052 T69 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 27052 T69 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 27054 T69 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 27056 T24 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 27056 T24 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 27057 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 27142 T24 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 27144 T24 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45848
[junit4:junit4] 2> 27144 T24 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 27145 T24 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 27145 T24 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371495611905
[junit4:junit4] 2> 27146 T24 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371495611905/solr.xml
[junit4:junit4] 2> 27146 T24 oasc.CoreContainer.<init> New CoreContainer 20464005
[junit4:junit4] 2> 27146 T24 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371495611905/'
[junit4:junit4] 2> 27147 T24 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371495611905/'
[junit4:junit4] 2> 27206 T24 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 27206 T24 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 27207 T24 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 27207 T24 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 27207 T24 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 27208 T24 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 27208 T24 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 27208 T24 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 27208 T24 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 27209 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 27213 T24 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 27213 T24 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:32827/solr
[junit4:junit4] 2> 27213 T24 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 27214 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 27216 T82 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@192eea0 name:ZooKeeperConnection Watcher:127.0.0.1:32827 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 27216 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 27218 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 27222 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 27224 T84 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1277b77 name:ZooKeeperConnection Watcher:127.0.0.1:32827/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 27224 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 27229 T24 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 28234 T24 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45848_ge
[junit4:junit4] 2> 28235 T24 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45848_ge
[junit4:junit4] 2> 28239 T48 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 28239 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 28239 T55 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 28240 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 28240 T68 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 28240 T84 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 28240 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 28249 T85 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 28249 T85 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 28549 T49 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 28551 T49 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46820/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:46820_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 28556 T49 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45848/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:45848_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 28556 T49 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 28556 T49 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 28562 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 28562 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 28562 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 28562 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 29251 T85 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371495611905/collection1
[junit4:junit4] 2> 29251 T85 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 29252 T85 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 29252 T85 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 29254 T85 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371495611905/collection1/'
[junit4:junit4] 2> 29254 T85 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371495611905/collection1/lib/README' to classloader
[junit4:junit4] 2> 29255 T85 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371495611905/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 29285 T85 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 29322 T85 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 29324 T85 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 29332 T85 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 29873 T85 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 29882 T85 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 29888 T85 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 29919 T85 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 29923 T85 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 29928 T85 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 29929 T85 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 29930 T85 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 29930 T85 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 29933 T85 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 29933 T85 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 29934 T85 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 29934 T85 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371495611905/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty2/
[junit4:junit4] 2> 29934 T85 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1b67c33
[junit4:junit4] 2> 29935 T85 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 29935 T85 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty2
[junit4:junit4] 2> 29936 T85 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty2/index/
[junit4:junit4] 2> 29936 T85 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 29937 T85 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty2/index
[junit4:junit4] 2> 29943 T85 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 29943 T85 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 29947 T85 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 29947 T85 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 29948 T85 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 29949 T85 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 29949 T85 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 29950 T85 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 29950 T85 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 29951 T85 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 29951 T85 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 29954 T85 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 29958 T85 oass.SolrIndexSearcher.<init> Opening Searcher@1781b5d main
[junit4:junit4] 2> 29959 T85 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 29959 T85 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 29966 T86 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1781b5d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 29971 T85 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 29971 T85 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45848/ge collection:collection1 shard:shard1
[junit4:junit4] 2> 29972 T85 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 29982 T85 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 29986 T85 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 29986 T85 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 29986 T85 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45848/ge/collection1/
[junit4:junit4] 2> 29987 T85 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 29987 T85 oasc.SyncStrategy.syncToMe http://127.0.0.1:45848/ge/collection1/ has no replicas
[junit4:junit4] 2> 29987 T85 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45848/ge/collection1/
[junit4:junit4] 2> 29987 T85 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 30071 T49 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 30082 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 30082 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 30082 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 30082 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 30095 T85 oasc.ZkController.register We are http://127.0.0.1:45848/ge/collection1/ and leader is http://127.0.0.1:45848/ge/collection1/
[junit4:junit4] 2> 30095 T85 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45848/ge
[junit4:junit4] 2> 30095 T85 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 30095 T85 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 30096 T85 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 30098 T85 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 30100 T24 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 30100 T24 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 30101 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 30197 T24 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 30200 T24 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43931
[junit4:junit4] 2> 30201 T24 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 30201 T24 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 30202 T24 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371495614950
[junit4:junit4] 2> 30202 T24 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371495614950/solr.xml
[junit4:junit4] 2> 30202 T24 oasc.CoreContainer.<init> New CoreContainer 24194761
[junit4:junit4] 2> 30203 T24 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371495614950/'
[junit4:junit4] 2> 30203 T24 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371495614950/'
[junit4:junit4] 2> 30261 T24 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 30262 T24 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 30262 T24 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 30262 T24 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 30263 T24 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 30263 T24 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 30263 T24 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 30264 T24 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 30264 T24 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 30264 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 30268 T24 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 30268 T24 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:32827/solr
[junit4:junit4] 2> 30269 T24 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 30270 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 30272 T98 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@974fda name:ZooKeeperConnection Watcher:127.0.0.1:32827 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 30272 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 30274 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 30278 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 30279 T100 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@762282 name:ZooKeeperConnection Watcher:127.0.0.1:32827/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 30280 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 30285 T24 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 31289 T24 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43931_ge
[junit4:junit4] 2> 31291 T24 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43931_ge
[junit4:junit4] 2> 31293 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 31293 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 31294 T68 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 31294 T48 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 31295 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 31295 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 31296 T100 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 31297 T55 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 31298 T84 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 31303 T101 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 31304 T101 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 31590 T49 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 31591 T49 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45848/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:45848_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 31596 T49 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43931/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:43931_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 31596 T49 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 31596 T49 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 31601 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 31601 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 31601 T100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 31601 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 31601 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 32305 T101 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371495614950/collection1
[junit4:junit4] 2> 32306 T101 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 32307 T101 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 32307 T101 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 32309 T101 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371495614950/collection1/'
[junit4:junit4] 2> 32310 T101 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371495614950/collection1/lib/README' to classloader
[junit4:junit4] 2> 32310 T101 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371495614950/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 32349 T101 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 32387 T101 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 32389 T101 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 32397 T101 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 32753 T101 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 32761 T101 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 32764 T101 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 32791 T101 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 32795 T101 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 32798 T101 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 32799 T101 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 32799 T101 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 32800 T101 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 32801 T101 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 32801 T101 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 32801 T101 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 32801 T101 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371495614950/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty3/
[junit4:junit4] 2> 32802 T101 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1b67c33
[junit4:junit4] 2> 32802 T101 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 32802 T101 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty3
[junit4:junit4] 2> 32803 T101 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty3/index/
[junit4:junit4] 2> 32803 T101 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 32803 T101 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty3/index
[junit4:junit4] 2> 32807 T101 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 32807 T101 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 32810 T101 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 32810 T101 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 32810 T101 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 32811 T101 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 32812 T101 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 32812 T101 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 32812 T101 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 32813 T101 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 32813 T101 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 32815 T101 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 32817 T101 oass.SolrIndexSearcher.<init> Opening Searcher@765443 main
[junit4:junit4] 2> 32818 T101 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 32818 T101 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 32825 T102 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@765443 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 32829 T101 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 32829 T101 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43931/ge collection:collection1 shard:shard2
[junit4:junit4] 2> 32830 T101 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 32837 T101 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 32840 T101 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 32840 T101 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 32840 T101 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43931/ge/collection1/
[junit4:junit4] 2> 32841 T101 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 32841 T101 oasc.SyncStrategy.syncToMe http://127.0.0.1:43931/ge/collection1/ has no replicas
[junit4:junit4] 2> 32841 T101 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43931/ge/collection1/
[junit4:junit4] 2> 32841 T101 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 33109 T49 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 33118 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 33118 T100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 33118 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 33118 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 33118 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 33152 T101 oasc.ZkController.register We are http://127.0.0.1:43931/ge/collection1/ and leader is http://127.0.0.1:43931/ge/collection1/
[junit4:junit4] 2> 33152 T101 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43931/ge
[junit4:junit4] 2> 33152 T101 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 33152 T101 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 33153 T101 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 33155 T101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 33156 T24 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 33157 T24 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 33157 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 33263 T24 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 33267 T24 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60099
[junit4:junit4] 2> 33268 T24 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 33268 T24 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 33269 T24 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371495618006
[junit4:junit4] 2> 33269 T24 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371495618006/solr.xml
[junit4:junit4] 2> 33269 T24 oasc.CoreContainer.<init> New CoreContainer 21270901
[junit4:junit4] 2> 33270 T24 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371495618006/'
[junit4:junit4] 2> 33270 T24 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371495618006/'
[junit4:junit4] 2> 33333 T24 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 33333 T24 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 33334 T24 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 33334 T24 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 33334 T24 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 33335 T24 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 33335 T24 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 33335 T24 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 33335 T24 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 33336 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 33339 T24 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 33340 T24 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:32827/solr
[junit4:junit4] 2> 33340 T24 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 33341 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 33343 T114 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1afdaef name:ZooKeeperConnection Watcher:127.0.0.1:32827 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 33343 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 33346 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 33349 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 33352 T116 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bbb286 name:ZooKeeperConnection Watcher:127.0.0.1:32827/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 33352 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 33358 T24 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 34362 T24 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60099_ge
[junit4:junit4] 2> 34363 T24 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60099_ge
[junit4:junit4] 2> 34366 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 34366 T100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 34367 T84 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 34367 T68 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 34367 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34367 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34367 T48 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 34368 T116 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 34368 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34369 T55 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 34370 T100 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 34375 T117 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 34375 T117 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 34627 T49 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 34628 T49 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43931/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:43931_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 34632 T49 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60099/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:60099_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 34633 T49 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 34633 T49 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 34638 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34638 T100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34638 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34638 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34638 T116 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34638 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 35377 T117 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371495618006/collection1
[junit4:junit4] 2> 35377 T117 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 35378 T117 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 35378 T117 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 35380 T117 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371495618006/collection1/'
[junit4:junit4] 2> 35380 T117 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371495618006/collection1/lib/README' to classloader
[junit4:junit4] 2> 35380 T117 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371495618006/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 35413 T117 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 35447 T117 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 35449 T117 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 35461 T117 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 35807 T117 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 35815 T117 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 35818 T117 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 35842 T117 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 35845 T117 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 35852 T117 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 35854 T117 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 35854 T117 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 35855 T117 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 35856 T117 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 35856 T117 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 35856 T117 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 35856 T117 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371495618006/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty4/
[junit4:junit4] 2> 35857 T117 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1b67c33
[junit4:junit4] 2> 35857 T117 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 35857 T117 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty4
[junit4:junit4] 2> 35858 T117 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty4/index/
[junit4:junit4] 2> 35858 T117 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 35858 T117 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty4/index
[junit4:junit4] 2> 35861 T117 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 35861 T117 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 35863 T117 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 35864 T117 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 35864 T117 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 35865 T117 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 35865 T117 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 35866 T117 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 35866 T117 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 35866 T117 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 35867 T117 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 35868 T117 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 35871 T117 oass.SolrIndexSearcher.<init> Opening Searcher@6d6ca5 main
[junit4:junit4] 2> 35872 T117 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 35872 T117 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 35879 T118 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6d6ca5 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 35883 T117 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 35883 T117 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60099/ge collection:collection1 shard:shard3
[junit4:junit4] 2> 35887 T117 oasc.ZkController.register We are http://127.0.0.1:60099/ge/collection1/ and leader is http://127.0.0.1:46820/ge/collection1/
[junit4:junit4] 2> 35887 T117 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60099/ge
[junit4:junit4] 2> 35888 T117 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 35888 T117 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C28 name=collection1 org.apache.solr.core.SolrCore@19548 url=http://127.0.0.1:60099/ge/collection1 node=127.0.0.1:60099_ge C28_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:60099/ge, node_name=127.0.0.1:60099_ge, core=collection1, state=down, collection=collection1}
[junit4:junit4] 2> 35893 T119 C28 P60099 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 35893 T119 C28 P60099 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 35893 T117 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 35894 T119 C28 P60099 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 35894 T119 C28 P60099 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 35895 T24 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 35896 T24 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 35895 T119 C28 P60099 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 35896 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 35985 T24 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 35988 T24 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55405
[junit4:junit4] 2> 35989 T24 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 35989 T24 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 35990 T24 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371495620746
[junit4:junit4] 2> 35990 T24 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371495620746/solr.xml
[junit4:junit4] 2> 35990 T24 oasc.CoreContainer.<init> New CoreContainer 32214115
[junit4:junit4] 2> 35991 T24 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371495620746/'
[junit4:junit4] 2> 35991 T24 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371495620746/'
[junit4:junit4] 2> 36050 T24 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 36051 T24 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 36051 T24 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 36052 T24 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 36052 T24 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 36052 T24 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 36053 T24 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 36053 T24 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 36054 T24 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 36055 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 36060 T24 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 36062 T24 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:32827/solr
[junit4:junit4] 2> 36062 T24 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 36063 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 36065 T131 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bc489d name:ZooKeeperConnection Watcher:127.0.0.1:32827 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 36065 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 36068 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 36073 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 36075 T133 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@166a0b name:ZooKeeperConnection Watcher:127.0.0.1:32827/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 36075 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 36080 T24 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 36147 T49 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 36149 T49 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60099/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:60099_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 36152 T62 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 36156 T100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 36166 T116 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 36166 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 36166 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 36156 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 36156 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 36156 T133 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 37085 T24 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55405_ge
[junit4:junit4] 2> 37088 T24 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55405_ge
[junit4:junit4] 2> 37090 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 37091 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 37091 T100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 37092 T133 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 37091 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 37090 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 37092 T133 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 37092 T116 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 37094 T116 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 37096 T55 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 37098 T100 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 37098 T84 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 37098 T68 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 37099 T48 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 37108 T134 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 37108 T134 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 37153 T62 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 37154 T62 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={onlyIfLeader=true&wt=javabin&nodeName=127.0.0.1:60099_ge&checkLive=true&version=2&coreNodeName=5&action=PREPRECOVERY&state=recovering&core=collection1} status=0 QTime=1003
[junit4:junit4] 2> 37675 T49 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 37676 T49 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55405/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:55405_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 37677 T49 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 37677 T49 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 37683 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 37683 T100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 37683 T133 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 37683 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 37683 T116 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 37683 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 37683 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 38110 T134 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371495620746/collection1
[junit4:junit4] 2> 38110 T134 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 38111 T134 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 38111 T134 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 38113 T134 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371495620746/collection1/'
[junit4:junit4] 2> 38113 T134 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371495620746/collection1/lib/README' to classloader
[junit4:junit4] 2> 38114 T134 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371495620746/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 38154 T134 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 38195 T134 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 38196 T134 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 38201 T134 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 38572 T134 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 38579 T134 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 38581 T134 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 38608 T134 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 38612 T134 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 38616 T134 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 38618 T134 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 38619 T134 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 38619 T134 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 38620 T134 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 38620 T134 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 38621 T134 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 38621 T134 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371495620746/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty5/
[junit4:junit4] 2> 38621 T134 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1b67c33
[junit4:junit4] 2> 38621 T134 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 38622 T134 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty5
[junit4:junit4] 2> 38622 T134 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty5/index/
[junit4:junit4] 2> 38622 T134 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 38623 T134 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty5/index
[junit4:junit4] 2> 38625 T134 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 38626 T134 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 38628 T134 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 38628 T134 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 38628 T134 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 38629 T134 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 38629 T134 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 38629 T134 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 38630 T134 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 38630 T134 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 38631 T134 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 38632 T134 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 38635 T134 oass.SolrIndexSearcher.<init> Opening Searcher@181985b main
[junit4:junit4] 2> 38635 T134 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 38636 T134 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 38642 T135 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@181985b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 38645 T134 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 38646 T134 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55405/ge collection:collection1 shard:shard1
[junit4:junit4] 2> 38649 T134 oasc.ZkController.register We are http://127.0.0.1:55405/ge/collection1/ and leader is http://127.0.0.1:45848/ge/collection1/
[junit4:junit4] 2> 38650 T134 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55405/ge
[junit4:junit4] 2> 38650 T134 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 38650 T134 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C29 name=collection1 org.apache.solr.core.SolrCore@17d777e url=http://127.0.0.1:55405/ge/collection1 node=127.0.0.1:55405_ge C29_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:55405/ge, node_name=127.0.0.1:55405_ge, core=collection1, state=down, collection=collection1}
[junit4:junit4] 2> 38651 T136 C29 P55405 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 38652 T136 C29 P55405 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 38652 T134 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 38652 T136 C29 P55405 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 38652 T136 C29 P55405 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 38653 T24 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 38654 T24 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 38654 T136 C29 P55405 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 38654 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 38659 T78 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 38729 T24 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 38732 T24 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39250
[junit4:junit4] 2> 38733 T24 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 38733 T24 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 38733 T24 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371495623501
[junit4:junit4] 2> 38734 T24 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371495623501/solr.xml
[junit4:junit4] 2> 38734 T24 oasc.CoreContainer.<init> New CoreContainer 3838506
[junit4:junit4] 2> 38734 T24 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371495623501/'
[junit4:junit4] 2> 38735 T24 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371495623501/'
[junit4:junit4] 2> 38787 T24 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 38787 T24 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 38788 T24 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 38788 T24 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 38788 T24 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 38788 T24 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 38789 T24 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 38789 T24 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 38789 T24 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 38790 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 38793 T24 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 38794 T24 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:32827/solr
[junit4:junit4] 2> 38794 T24 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 38795 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 38797 T148 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@da5af4 name:ZooKeeperConnection Watcher:127.0.0.1:32827 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 38797 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 38799 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 38802 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 38804 T150 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@413db7 name:ZooKeeperConnection Watcher:127.0.0.1:32827/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 38804 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 38808 T24 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C30 name=collection1 org.apache.solr.core.SolrCore@19548 url=http://127.0.0.1:60099/ge/collection1 node=127.0.0.1:60099_ge C30_STATE=coll:collection1 core:collection1 props:{shard=shard3, base_url=http://127.0.0.1:60099/ge, node_name=127.0.0.1:60099_ge, core=collection1, state=recovering, collection=collection1}
[junit4:junit4] 2> 39184 T119 C30 P60099 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:46820/ge/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 39190 T119 C30 P60099 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 39192 T49 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 39193 T49 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55405/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:55405_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 39198 T119 C30 P60099 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:60099/ge START replicas=[http://127.0.0.1:46820/ge/collection1/] nUpdates=100
[junit4:junit4] 2> 39200 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 39201 T133 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 39201 T116 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 39201 T150 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 39201 T119 C30 P60099 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 39202 T119 C30 P60099 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 39202 T119 C30 P60099 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 39201 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 39201 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 39203 T119 C30 P60099 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 39203 T119 C30 P60099 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 39203 T119 C30 P60099 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:46820/ge/collection1/. core=collection1
[junit4:junit4] 2> 39201 T100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 39201 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 39204 T119 C30 P60099 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C31 name=collection1 org.apache.solr.core.SolrCore@d2a4a8 url=http://127.0.0.1:46820/ge/collection1 node=127.0.0.1:46820_ge C31_STATE=coll:collection1 core:collection1 props:{shard=shard3, base_url=http://127.0.0.1:46820/ge, node_name=127.0.0.1:46820_ge, core=collection1, state=active, collection=collection1, leader=true}
[junit4:junit4] 2> 39222 T63 C31 P46820 oasc.SolrCore.execute [collection1] webapp=/ge path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=2
[junit4:junit4] 2> 39224 T61 C31 P46820 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 39227 T61 C31 P46820 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 39227 T61 C31 P46820 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 39235 T61 C31 P46820 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 39235 T61 C31 P46820 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 39236 T61 C31 P46820 oass.SolrIndexSearcher.<init> Opening Searcher@18a3c1e realtime
[junit4:junit4] 2> 39236 T61 C31 P46820 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 39236 T61 C31 P46820 oasup.LogUpdateProcessor.finish [collection1] webapp=/ge path=/update params={waitSearcher=true&openSearcher=false&commit_end_point=true&wt=javabin&commit=true&version=2&softCommit=false} {commit=} 0 16
[junit4:junit4] 2> 39257 T119 C30 P60099 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 39257 T119 C30 P60099 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4] 2> 39262 T119 C30 P60099 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 39277 T62 C31 P46820 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 39277 T62 C31 P46820 oasc.SolrCore.execute [collection1] webapp=/ge path=/replication params={wt=javabin&command=indexversion&qt=/replication&version=2} status=0 QTime=13
[junit4:junit4] 2> 39278 T119 C30 P60099 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 39279 T119 C30 P60099 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 39279 T119 C30 P60099 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 39282 T63 C31 P46820 oasc.SolrCore.execute [collection1] webapp=/ge path=/replication params={wt=javabin&command=filelist&qt=/replication&version=2&generation=2} status=0 QTime=1
[junit4:junit4] 2> 39282 T119 C30 P60099 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 39283 T119 C30 P60099 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty4/index.20130617140024126
[junit4:junit4] 2> 39283 T119 C30 P60099 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty4/index.20130617140024126 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1955e48 fullCopy=false
[junit4:junit4] 2> ASYNC NEW_CORE C32 name=collection1 org.apache.solr.core.SolrCore@d2a4a8 url=http://127.0.0.1:46820/ge/collection1 node=127.0.0.1:46820_ge C32_STATE=coll:collection1 core:collection1 props:{shard=shard3, base_url=http://127.0.0.1:46820/ge, node_name=127.0.0.1:46820_ge, core=collection1, state=active, collection=collection1, leader=true}
[junit4:junit4] 2> 39296 T61 C32 P46820 oasc.SolrCore.execute [collection1] webapp=/ge path=/replication params={wt=filestream&command=filecontent&qt=/replication&checksum=true&file=segments_2&generation=2} status=0 QTime=2
[junit4:junit4] 2> ASYNC NEW_CORE C33 name=collection1 org.apache.solr.core.SolrCore@19548 url=http://127.0.0.1:60099/ge/collection1 node=127.0.0.1:60099_ge C33_STATE=coll:collection1 core:collection1 props:{shard=shard3, base_url=http://127.0.0.1:60099/ge, node_name=127.0.0.1:60099_ge, core=collection1, state=recovering, collection=collection1}
[junit4:junit4] 2> 39302 T119 C33 P60099 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 39308 T119 C33 P60099 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 39309 T119 C33 P60099 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 39313 T119 C33 P60099 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 39313 T119 C33 P60099 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 39314 T119 C33 P60099 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 39314 T119 C33 P60099 oass.SolrIndexSearcher.<init> Opening Searcher@38f011 main
[junit4:junit4] 2> 39316 T118 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@38f011 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 39316 T119 C33 P60099 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty4/index.20130617140024126 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty4/index.20130617140024126;done=true>>]
[junit4:junit4] 2> 39317 T119 C33 P60099 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty4/index.20130617140024126
[junit4:junit4] 2> 39317 T119 C33 P60099 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty4/index.20130617140024126
[junit4:junit4] 2> 39321 T119 C33 P60099 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 39321 T119 C33 P60099 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 39321 T119 C33 P60099 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 39321 T119 C33 P60099 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 39323 T119 C33 P60099 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 39660 T78 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 39660 T78 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={onlyIfLeader=true&wt=javabin&nodeName=127.0.0.1:55405_ge&checkLive=true&version=2&coreNodeName=6&action=PREPRECOVERY&state=recovering&core=collection1} status=0 QTime=1001
[junit4:junit4] 2> 39812 T24 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39250_ge
[junit4:junit4] 2> 39813 T24 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39250_ge
[junit4:junit4] 2> 39816 T150 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 39816 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 39816 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 39816 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 39816 T133 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 39817 T100 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 39817 T116 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 39817 T116 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 39816 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 39817 T100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 39820 T55 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 39820 T68 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 39820 T150 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 39820 T84 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 39820 T133 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 39821 T48 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 39826 T153 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 39827 T153 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 40712 T49 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 40714 T49 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60099/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:60099_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 40720 T49 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39250/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:39250_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 40720 T49 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 40720 T49 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 40739 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 40739 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 40740 T150 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 40740 T133 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 40740 T100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 40740 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 40740 T116 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 40740 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 40828 T153 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371495623501/collection1
[junit4:junit4] 2> 40828 T153 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 40829 T153 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 40829 T153 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 40831 T153 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371495623501/collection1/'
[junit4:junit4] 2> 40831 T153 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371495623501/collection1/lib/README' to classloader
[junit4:junit4] 2> 40831 T153 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371495623501/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 40867 T153 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 40902 T153 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 40904 T153 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 40908 T153 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 41260 T153 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 41267 T153 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 41273 T153 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 41295 T153 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 41300 T153 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 41305 T153 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 41307 T153 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 41307 T153 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 41307 T153 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 41309 T153 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 41310 T153 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 41310 T153 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 41310 T153 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371495623501/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty6/
[junit4:junit4] 2> 41311 T153 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1b67c33
[junit4:junit4] 2> 41311 T153 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 41312 T153 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty6
[junit4:junit4] 2> 41312 T153 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty6/index/
[junit4:junit4] 2> 41313 T153 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 41313 T153 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty6/index
[junit4:junit4] 2> 41317 T153 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 41318 T153 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 41322 T153 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 41323 T153 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 41324 T153 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 41325 T153 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 41326 T153 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 41326 T153 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 41327 T153 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 41328 T153 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 41329 T153 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 41332 T153 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 41337 T153 oass.SolrIndexSearcher.<init> Opening Searcher@14b29a2 main
[junit4:junit4] 2> 41338 T153 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 41338 T153 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 41343 T154 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@14b29a2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 41346 T153 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 41346 T153 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39250/ge collection:collection1 shard:shard2
[junit4:junit4] 2> 41349 T153 oasc.ZkController.register We are http://127.0.0.1:39250/ge/collection1/ and leader is http://127.0.0.1:43931/ge/collection1/
[junit4:junit4] 2> 41349 T153 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39250/ge
[junit4:junit4] 2> 41349 T153 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 41349 T153 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C34 name=collection1 org.apache.solr.core.SolrCore@7e55c5 url=http://127.0.0.1:39250/ge/collection1 node=127.0.0.1:39250_ge C34_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:39250/ge, node_name=127.0.0.1:39250_ge, core=collection1, state=down, collection=collection1}
[junit4:junit4] 2> 41350 T155 C34 P39250 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 41350 T155 C34 P39250 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 41350 T155 C34 P39250 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 41351 T155 C34 P39250 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 41350 T153 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 41351 T155 C34 P39250 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 41352 T24 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 41352 T24 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 41353 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 41356 T94 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 41424 T24 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 41426 T24 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60894
[junit4:junit4] 2> 41426 T24 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 41427 T24 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 41427 T24 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371495626199
[junit4:junit4] 2> 41427 T24 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371495626199/solr.xml
[junit4:junit4] 2> 41428 T24 oasc.CoreContainer.<init> New CoreContainer 17774515
[junit4:junit4] 2> 41428 T24 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371495626199/'
[junit4:junit4] 2> 41428 T24 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371495626199/'
[junit4:junit4] 2> 41480 T24 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 41480 T24 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 41481 T24 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 41481 T24 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 41481 T24 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 41481 T24 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 41482 T24 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 41482 T24 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 41482 T24 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 41483 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 41486 T24 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 41486 T24 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:32827/solr
[junit4:junit4] 2> 41487 T24 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 41488 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 41489 T167 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1565628 name:ZooKeeperConnection Watcher:127.0.0.1:32827 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 41490 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 41492 T24 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 41495 T24 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 41497 T169 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@543c60 name:ZooKeeperConnection Watcher:127.0.0.1:32827/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 41497 T24 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 41502 T24 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C35 name=collection1 org.apache.solr.core.SolrCore@17d777e url=http://127.0.0.1:55405/ge/collection1 node=127.0.0.1:55405_ge C35_STATE=coll:collection1 core:collection1 props:{shard=shard1, base_url=http://127.0.0.1:55405/ge, node_name=127.0.0.1:55405_ge, core=collection1, state=recovering, collection=collection1}
[junit4:junit4] 2> 41661 T136 C35 P55405 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:45848/ge/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 41662 T136 C35 P55405 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55405/ge START replicas=[http://127.0.0.1:45848/ge/collection1/] nUpdates=100
[junit4:junit4] 2> 41662 T136 C35 P55405 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 41662 T136 C35 P55405 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 41662 T136 C35 P55405 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 41663 T136 C35 P55405 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 41663 T136 C35 P55405 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 41663 T136 C35 P55405 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:45848/ge/collection1/. core=collection1
[junit4:junit4] 2> 41663 T136 C35 P55405 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C36 name=collection1 org.apache.solr.core.SolrCore@18d3da1 url=http://127.0.0.1:45848/ge/collection1 node=127.0.0.1:45848_ge C36_STATE=coll:collection1 core:collection1 props:{shard=shard1, base_url=http://127.0.0.1:45848/ge, node_name=127.0.0.1:45848_ge, core=collection1, state=active, collection=collection1, leader=true}
[junit4:junit4] 2> 41668 T78 C36 P45848 oasc.SolrCore.execute [collection1] webapp=/ge path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 41668 T77 C36 P45848 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 41670 T77 C36 P45848 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 41671 T77 C36 P45848 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 41675 T77 C36 P45848 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 41675 T77 C36 P45848 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 41676 T77 C36 P45848 oass.SolrIndexSearcher.<init> Opening Searcher@172f35c realtime
[junit4:junit4] 2> 41676 T77 C36 P45848 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 41676 T77 C36 P45848 oasup.LogUpdateProcessor.finish [collection1] webapp=/ge path=/update params={waitSearcher=true&openSearcher=false&commit_end_point=true&wt=javabin&commit=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 41677 T136 C35 P55405 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 41678 T136 C35 P55405 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 41680 T78 C36 P45848 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 41680 T78 C36 P45848 oasc.SolrCore.execute [collection1] webapp=/ge path=/replication params={wt=javabin&command=indexversion&qt=/replication&version=2} status=0 QTime=0
[junit4:junit4] 2> 41681 T136 C35 P55405 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 41681 T136 C35 P55405 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 41681 T136 C35 P55405 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 41684 T77 C36 P45848 oasc.SolrCore.execute [collection1] webapp=/ge path=/replication params={wt=javabin&command=filelist&qt=/replication&version=2&generation=2} status=0 QTime=1
[junit4:junit4] 2> 41684 T136 C35 P55405 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 41685 T136 C35 P55405 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty5/index.20130617140026528
[junit4:junit4] 2> 41685 T136 C35 P55405 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty5/index.20130617140026528 lockFactory=org.apache.lucene.store.NativeFSLockFactory@bf19f8 fullCopy=false
[junit4:junit4] 2> 41688 T79 C36 P45848 oasc.SolrCore.execute [collection1] webapp=/ge path=/replication params={wt=filestream&command=filecontent&qt=/replication&checksum=true&file=segments_2&generation=2} status=0 QTime=0
[junit4:junit4] 2> 41689 T136 C35 P55405 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 41695 T136 C35 P55405 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 41696 T136 C35 P55405 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 41697 T136 C35 P55405 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty5/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 41698 T136 C35 P55405 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 41698 T136 C35 P55405 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 41698 T136 C35 P55405 oass.SolrIndexSearcher.<init> Opening Searcher@19e078b main
[junit4:junit4] 2> 41699 T135 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19e078b main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 41700 T136 C35 P55405 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty5/index.20130617140026528 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty5/index.20130617140026528;done=true>>]
[junit4:junit4] 2> 41700 T136 C35 P55405 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty5/index.20130617140026528
[junit4:junit4] 2> 41700 T136 C35 P55405 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty5/index.20130617140026528
[junit4:junit4] 2> 41701 T136 C35 P55405 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 41701 T136 C35 P55405 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 41701 T136 C35 P55405 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 41701 T136 C35 P55405 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 41725 T136 C35 P55405 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 42248 T49 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 42249 T49 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39250/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:39250_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 42253 T49 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55405/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:55405_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 42257 T169 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 42257 T150 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 42258 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 42258 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 42259 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 42258 T133 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 42257 T100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 42258 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 42258 T116 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 42357 T94 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 42357 T94 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={onlyIfLeader=true&wt=javabin&nodeName=127.0.0.1:39250_ge&checkLive=true&version=2&coreNodeName=7&action=PREPRECOVERY&state=recovering&core=collection1} status=0 QTime=1001
[junit4:junit4] 2> 42506 T24 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60894_ge
[junit4:junit4] 2> 42507 T24 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60894_ge
[junit4:junit4] 2> 42509 T133 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 42509 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 42510 T150 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 42510 T169 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 42511 T55 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 42511 T84 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 42511 T116 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 42511 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 42511 T68 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 42512 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 42511 T100 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 42511 T116 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 42511 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 42512 T100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 42514 T133 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 42514 T48 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 42515 T169 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 42515 T150 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 42529 T172 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 42530 T172 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 43769 T49 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 43770 T49 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "core_node_name":"8",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60894/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:60894_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 43770 T49 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 43771 T49 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 43777 T133 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 43778 T84 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 43778 T68 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 43778 T169 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 43778 T100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 43778 T150 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 43778 T116 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 43778 T55 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 43778 T48 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> ASYNC NEW_CORE C37 name=collection1 org.apache.solr.core.SolrCore@7e55c5 url=http://127.0.0.1:39250/ge/collection1 node=127.0.0.1:39250_ge C37_STATE=coll:collection1 core:collection1 props:{shard=shard2, base_url=http://127.0.0.1:39250/ge, node_name=127.0.0.1:39250_ge, core=collection1, state=recovering, collection=collection1}
[junit4:junit4] 2> 44358 T155 C37 P39250 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:43931/ge/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 44359 T155 C37 P39250 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39250/ge START replicas=[http://127.0.0.1:43931/ge/collection1/] nUpdates=100
[junit4:junit4] 2> 44359 T155 C37 P39250 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 44359 T155 C37 P39250 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 44360 T155 C37 P39250 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 44360 T155 C37 P39250 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 44360 T155 C37 P39250 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 44360 T155 C37 P39250 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:43931/ge/collection1/. core=collection1
[junit4:junit4] 2> 44360 T155 C37 P39250 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C38 name=collection1 org.apache.solr.core.SolrCore@340296 url=http://127.0.0.1:43931/ge/collection1 node=127.0.0.1:43931_ge C38_STATE=coll:collection1 core:collection1 props:{shard=shard2, base_url=http://127.0.0.1:43931/ge, node_name=127.0.0.1:43931_ge, core=collection1, state=active, collection=collection1, leader=true}
[junit4:junit4] 2> 44364 T94 C38 P43931 oasc.SolrCore.execute [collection1] webapp=/ge path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 44366 T93 C38 P43931 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 44368 T93 C38 P43931 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 44368 T93 C38 P43931 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 44375 T93 C38 P43931 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty3/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 44375 T93 C38 P43931 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 44376 T93 C38 P43931 oass.SolrIndexSearcher.<init> Opening Searcher@145d360 realtime
[junit4:junit4] 2> 44376 T93 C38 P43931 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 44376 T93 C38 P43931 oasup.LogUpdateProcessor.finish [collection1] webapp=/ge path=/update params={waitSearcher=true&openSearcher=false&commit_end_point=true&wt=javabin&commit=true&version=2&softCommit=false} {commit=} 0 10
[junit4:junit4] 2> 44377 T155 C37 P39250 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 44377 T155 C37 P39250 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 44379 T95 C38 P43931 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 44379 T95 C38 P43931 oasc.SolrCore.execute [collection1] webapp=/ge path=/replication params={wt=javabin&command=indexversion&qt=/replication&version=2} status=0 QTime=0
[junit4:junit4] 2> 44380 T155 C37 P39250 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 44380 T155 C37 P39250 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 44380 T155 C37 P39250 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 44382 T94 C38 P43931 oasc.SolrCore.execute [collection1] webapp=/ge path=/replication params={wt=javabin&command=filelist&qt=/replication&version=2&generation=2} status=0 QTime=0
[junit4:junit4] 2> 44383 T155 C37 P39250 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 44383 T155 C37 P39250 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty6/index.20130617140029226
[junit4:junit4] 2> 44384 T155 C37 P39250 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty6/index.20130617140029226 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1ec1cf4 fullCopy=false
[junit4:junit4] 2> 44386 T93 C38 P43931 oasc.SolrCore.execute [collection1] webapp=/ge path=/replication params={wt=filestream&command=filecontent&qt=/replication&checksum=true&file=segments_2&generation=2} status=0 QTime=0
[junit4:junit4] 2> 44388 T155 C37 P39250 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 44395 T155 C37 P39250 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 44396 T155 C37 P39250 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 44398 T155 C37 P39250 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty6/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 44398 T155 C37 P39250 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 44398 T155 C37 P39250 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 44399 T155 C37 P39250 oass.SolrIndexSearcher.<init> Opening Searcher@49d1d4 main
[junit4:junit4] 2> 44400 T154 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@49d1d4 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 44400 T155 C37 P39250 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty6/index.20130617140029226 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty6/index.20130617140029226;done=true>>]
[junit4:junit4] 2> 44401 T155 C37 P39250 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty6/index.20130617140029226
[junit4:junit4] 2> 44401 T155 C37 P39250 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty6/index.20130617140029226
[junit4:junit4] 2> 44401 T155 C37 P39250 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 44401 T155 C37 P39250 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 44401 T155 C37 P39250 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 44402 T155 C37 P39250 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 44415 T155 C37 P39250 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 44531 T172 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371495626199/collection1
[junit4:junit4] 2> 44531 T172 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 44532 T172 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 44532 T172 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 44534 T172 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371495626199/collection1/'
[junit4:junit4] 2> 44535 T172 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371495626199/collection1/lib/README' to classloader
[junit4:junit4] 2> 44535 T172 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371495626199/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 44580 T172 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 44621 T172 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 44623 T172 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 44627 T172 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 44988 T172 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 44998 T172 oass.IndexSchema.readSchema default search field
[...truncated too long message...]
.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4] 2> 230779 T247 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 230800 T24 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/ge,null}
[junit4:junit4] 2> 230851 T24 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 59685
[junit4:junit4] 2> 230851 T24 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=22391577
[junit4:junit4] 2> 230986 T343 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 230987 T343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 230987 T343 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 231647 T498 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 231648 T498 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "core_node_name":"11",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "numShards":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54784/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:54784_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 231651 T498 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "core_node_name":"12",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59685/ge",
[junit4:junit4] 2> "node_name":"127.0.0.1:59685_ge",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "collection":"collection1"}
[junit4:junit4] 2> 231654 T247 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 231853 T24 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 231853 T24 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 231855 T24 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@12590f7
[junit4:junit4] 2> 231856 T24 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=1276,cumulative_deletesById=642,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 231856 T24 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 231856 T24 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 231856 T24 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 231858 T24 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 231858 T24 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 231858 T24 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty11/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty11/index;done=false>>]
[junit4:junit4] 2> 231859 T24 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty11/index
[junit4:junit4] 2> 231859 T24 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty11 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty11;done=false>>]
[junit4:junit4] 2> 231859 T24 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371495604517/jetty11
[junit4:junit4] 2> 231860 T498 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89882335942737946-127.0.0.1:59685_ge-n_0000000011) am no longer a leader.
[junit4:junit4] 2> 231883 T24 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/ge,null}
[junit4:junit4] 2> 231934 T24 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 37410
[junit4:junit4] 2> 231964 T24 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 231967 T24 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:32827 32827
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=472419F524A213F6 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr_CS -Dtests.timezone=America/Rainy_River -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE 212s J1 | ChaosMonkeySafeLeaderTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 634 from http://127.0.0.1:45848/ge/collection1lastClient and got 689 from http://127.0.0.1:55405/ge/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([472419F524A213F6:C6C297ED53FD73CA]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 232005 T24 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 212351 T23 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 233161 T247 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 233161 T247 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 233162 T247 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {intDefault=Lucene41(blocksize=128), _version_=Lucene41(blocksize=128), timestamp=Lucene41(blocksize=128), multiDefault=PostingsFormat(name=Lucene41WithOrds), other_tl1=Lucene41(blocksize=128), a_si=PostingsFormat(name=Lucene41WithOrds), range_facet_l=Lucene41(blocksize=128), range_facet_si=PostingsFormat(name=Lucene41WithOrds), a_t=Lucene41(blocksize=128), text=PostingsFormat(name=Direct), id=MockFixedIntBlock(blockSize=1381), range_facet_sl=PostingsFormat(name=Direct), rnd_b=MockFixedIntBlock(blockSize=1381)}, docValues:{timestamp=DocValuesFormat(name=SimpleText)}, sim=DefaultSimilarity, locale=sr_CS, timezone=America/Rainy_River
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation 1.8.0-ea (32-bit)/cpus=8,threads=2,free=37118320,total=113868800
[junit4:junit4] 2> NOTE: All tests run in this JVM: [BadIndexSchemaTest, RequiredFieldsTest, TestRangeQuery, TestMergePolicyConfig, TestArbitraryIndexDir, ChaosMonkeySafeLeaderTest]
[junit4:junit4] Completed on J1 in 213.64s, 1 test, 1 failure <<< FAILURES!
[...truncated 755 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1249 tests, 1 failure, 16 ignored (7 assumptions)
Total time: 38 minutes 0 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-ea-b93 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure