You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2013/05/15 12:26:08 UTC
[JENKINS] Lucene-Solr-Tests-trunk-java7 - Build # 3984 - Still
Failing
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-java7/3984/
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeyShardSplitTest.testDistribSearch
Error Message:
Server at http://127.0.0.1:15017/ofu returned non ok status:500, message:Server Error
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:15017/ofu returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([4E27E2E9750C1754:CFC16CF102537768]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9327 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeyShardSplitTest
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.404; org.apache.solr.BaseDistributedSearchTestCase; Setting hostContext system property: /ofu/
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.410; org.apache.solr.SolrTestCaseJ4; ###Starting testDistribSearch
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.411; org.apache.solr.cloud.ZkTestServer; STARTING ZK TEST SERVER
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.412; org.apache.solr.cloud.ZkTestServer$ZKServerMain; Starting server
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.513; org.apache.solr.cloud.ZkTestServer; start zk server on port:20553
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.514; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.521; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@35b51c18 name:ZooKeeperConnection Watcher:127.0.0.1:20553 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.521; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.522; org.apache.solr.common.cloud.SolrZkClient; makePath: /solr
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.577; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.579; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@31eeda9d name:ZooKeeperConnection Watcher:127.0.0.1:20553/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.579; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.580; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.621; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/collection1/shards
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.656; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/control_collection
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.660; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/control_collection/shards
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.664; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.665; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.680; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.681; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/schema.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.786; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.787; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.799; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.800; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/protwords.txt
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.804; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.805; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/currency.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.810; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.810; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.815; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.816; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.821; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.821; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.826; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 1> INFO - 2013-05-15 09:57:41.827; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.047; org.eclipse.jetty.server.Server; jetty-8.1.10.v20130312
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.053; org.eclipse.jetty.server.AbstractConnector; Started SocketConnector@127.0.0.1:23494
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.054; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init()
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.055; org.apache.solr.core.SolrResourceLoader; JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.056; org.apache.solr.core.SolrResourceLoader; using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368611861833
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.056; org.apache.solr.core.CoreContainer$Initializer; looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368611861833/solr.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.057; org.apache.solr.core.CoreContainer; New CoreContainer 835866021
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.058; org.apache.solr.core.CoreContainer; Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368611861833/'
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.059; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368611861833/'
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.175; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting socketTimeout to: 120000
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.176; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting urlScheme to: http://
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.176; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting connTimeout to: 15000
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.177; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxConnectionsPerHost to: 20
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.177; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting corePoolSize to: 0
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.178; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maximumPoolSize to: 2147483647
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.179; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxThreadIdleTime to: 5
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.179; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting sizeOfQueue to: -1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.180; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting fairnessPolicy to: false
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.180; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.199; org.apache.solr.logging.LogWatcher; Registering Log Listener
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.200; org.apache.solr.core.ZkContainer; Zookeeper client=127.0.0.1:20553/solr
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.200; org.apache.solr.cloud.ZkController; zkHost includes chroot
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.202; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.231; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@52172cf3 name:ZooKeeperConnection Watcher:127.0.0.1:20553 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.231; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.234; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.249; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.251; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@2b67774d name:ZooKeeperConnection Watcher:127.0.0.1:20553/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.252; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.255; org.apache.solr.common.cloud.SolrZkClient; makePath: /overseer/queue
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.266; org.apache.solr.common.cloud.SolrZkClient; makePath: /overseer/collection-queue-work
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.272; org.apache.solr.common.cloud.SolrZkClient; makePath: /live_nodes
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.275; org.apache.solr.cloud.ZkController; Register node as live in ZooKeeper:/live_nodes/127.0.0.1:23494_ofu
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.277; org.apache.solr.common.cloud.SolrZkClient; makePath: /live_nodes/127.0.0.1:23494_ofu
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.288; org.apache.solr.common.cloud.SolrZkClient; makePath: /overseer_elect/election
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.303; org.apache.solr.common.cloud.SolrZkClient; makePath: /overseer_elect/leader
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.308; org.apache.solr.cloud.Overseer; Overseer (id=89693346949693443-127.0.0.1:23494_ofu-n_0000000000) starting
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.322; org.apache.solr.common.cloud.SolrZkClient; makePath: /overseer/queue-work
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.333; org.apache.solr.cloud.OverseerCollectionProcessor; Process current queue of collection creations
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.334; org.apache.solr.common.cloud.SolrZkClient; makePath: /clusterstate.json
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.339; org.apache.solr.common.cloud.SolrZkClient; makePath: /aliases.json
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.342; org.apache.solr.common.cloud.ZkStateReader; Updating cluster state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.346; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Starting to work on the main queue
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.351; org.apache.solr.core.CoreContainer; Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368611861833/collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.352; org.apache.solr.cloud.ZkController; Check for collection zkNode:control_collection
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.353; org.apache.solr.cloud.ZkController; Collection zkNode exists
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.353; org.apache.solr.cloud.ZkController; Load collection config from:/collections/control_collection
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.355; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368611861833/collection1/'
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.357; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368611861833/collection1/lib/classes/' to classloader
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.358; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368611861833/collection1/lib/README' to classloader
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.412; org.apache.solr.core.SolrConfig; Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.473; org.apache.solr.core.SolrConfig; Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.575; org.apache.solr.schema.IndexSchema; Reading Solr Schema from schema.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:42.582; org.apache.solr.schema.IndexSchema; [collection1] Schema name=test
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.219; org.apache.solr.schema.IndexSchema; default search field in schema is text
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.223; org.apache.solr.schema.IndexSchema; unique key field: id
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.225; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.232; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.287; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.288; org.apache.solr.core.SolrCore; [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1368611861833/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/control/data/
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.288; org.apache.solr.core.JmxMonitoredMap; JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@49df819
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.289; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.290; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/control/data
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.290; org.apache.solr.core.SolrCore; New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/control/data/index/
[junit4:junit4] 1> WARN - 2013-05-15 09:57:43.291; org.apache.solr.core.SolrCore; [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.291; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/control/data/index
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.296; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@18aae7e9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@76e1b416),segFN=segments_1,generation=1}
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.296; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.354; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.355; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.356; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.357; org.apache.solr.core.RequestHandlers; created standard: solr.StandardRequestHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.358; org.apache.solr.core.RequestHandlers; created /get: solr.RealTimeGetHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.358; org.apache.solr.core.RequestHandlers; adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.359; org.apache.solr.core.RequestHandlers; created /replication: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.359; org.apache.solr.core.RequestHandlers; created /update: solr.UpdateRequestHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.360; org.apache.solr.core.RequestHandlers; created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.373; org.apache.solr.handler.loader.XMLLoader; xsltCacheLifetimeSeconds=60
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.380; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@740f170e main
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.386; org.apache.solr.update.UpdateHandler; Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/control/data/tlog
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.407; org.apache.solr.update.CommitTracker; Hard AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.408; org.apache.solr.update.CommitTracker; Soft AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.413; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@740f170e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.416; org.apache.solr.cloud.ZkController; publishing core=collection1 state=down
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.416; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.853; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.854; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=1 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"1",
[junit4:junit4] 1> "numShards":"1",
[junit4:junit4] 1> "shard":null,
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:23494_ofu",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:23494/ofu"}
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.855; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Create collection control_collection with numShards 1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.856; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Assigning new node to shard=shard1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:43.871; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:44.419; org.apache.solr.core.CoreContainer; registering core: collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:44.419; org.apache.solr.cloud.ZkController; Register replica - core:collection1 address:http://127.0.0.1:23494/ofu collection:control_collection shard:shard1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:44.421; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 1> INFO - 2013-05-15 09:57:44.440; org.apache.solr.cloud.ShardLeaderElectionContext; Running the leader process.
[junit4:junit4] 1> INFO - 2013-05-15 09:57:44.450; org.apache.solr.cloud.ShardLeaderElectionContext; Enough replicas found to continue.
[junit4:junit4] 1> INFO - 2013-05-15 09:57:44.451; org.apache.solr.cloud.ShardLeaderElectionContext; I may be the new leader - try and sync
[junit4:junit4] 1> INFO - 2013-05-15 09:57:44.451; org.apache.solr.cloud.SyncStrategy; Sync replicas to http://127.0.0.1:23494/ofu/collection1/
[junit4:junit4] 1> INFO - 2013-05-15 09:57:44.452; org.apache.solr.cloud.SyncStrategy; Sync Success - now sync replicas to me
[junit4:junit4] 1> INFO - 2013-05-15 09:57:44.452; org.apache.solr.cloud.SyncStrategy; http://127.0.0.1:23494/ofu/collection1/ has no replicas
[junit4:junit4] 1> INFO - 2013-05-15 09:57:44.453; org.apache.solr.cloud.ShardLeaderElectionContext; I am the new leader: http://127.0.0.1:23494/ofu/collection1/
[junit4:junit4] 1> INFO - 2013-05-15 09:57:44.454; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.378; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.398; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.451; org.apache.solr.cloud.ZkController; We are http://127.0.0.1:23494/ofu/collection1/ and leader is http://127.0.0.1:23494/ofu/collection1/
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.456; org.apache.solr.cloud.ZkController; No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:23494/ofu
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.456; org.apache.solr.cloud.ZkController; I am the leader, no recovery necessary
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.457; org.apache.solr.cloud.ZkController; publishing core=collection1 state=active
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.457; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.460; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.462; org.apache.solr.servlet.SolrDispatchFilter; user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.462; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init() done
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.463; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.484; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.491; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.494; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@7b73e4d8 name:ZooKeeperConnection Watcher:127.0.0.1:20553/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.495; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.497; org.apache.solr.common.cloud.ZkStateReader; Updating cluster state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.504; org.apache.solr.cloud.ChaosMonkey; monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.744; org.eclipse.jetty.server.Server; jetty-8.1.10.v20130312
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.747; org.eclipse.jetty.server.AbstractConnector; Started SocketConnector@127.0.0.1:15017
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.749; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init()
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.749; org.apache.solr.core.SolrResourceLoader; JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.750; org.apache.solr.core.SolrResourceLoader; using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368611865504
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.750; org.apache.solr.core.CoreContainer$Initializer; looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368611865504/solr.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.751; org.apache.solr.core.CoreContainer; New CoreContainer 635226946
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.752; org.apache.solr.core.CoreContainer; Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368611865504/'
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.753; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368611865504/'
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.879; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting socketTimeout to: 120000
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.880; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting urlScheme to: http://
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.881; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting connTimeout to: 15000
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.881; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxConnectionsPerHost to: 20
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.882; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting corePoolSize to: 0
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.883; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maximumPoolSize to: 2147483647
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.883; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxThreadIdleTime to: 5
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.884; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting sizeOfQueue to: -1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.884; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting fairnessPolicy to: false
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.885; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.903; org.apache.solr.logging.LogWatcher; Registering Log Listener
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.904; org.apache.solr.core.ZkContainer; Zookeeper client=127.0.0.1:20553/solr
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.905; org.apache.solr.cloud.ZkController; zkHost includes chroot
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.906; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.909; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@7f1755c6 name:ZooKeeperConnection Watcher:127.0.0.1:20553 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.910; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.912; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.927; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.929; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@4d366eb1 name:ZooKeeperConnection Watcher:127.0.0.1:20553/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.929; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:45.943; org.apache.solr.common.cloud.ZkStateReader; Updating cluster state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.913; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.915; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=1 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"1",
[junit4:junit4] 1> "numShards":"1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:23494_ofu",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:23494/ofu"}
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.937; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.937; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.937; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.948; org.apache.solr.cloud.ZkController; Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15017_ofu
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.951; org.apache.solr.common.cloud.SolrZkClient; makePath: /live_nodes/127.0.0.1:15017_ofu
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.954; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.955; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (2)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.955; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (2)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.955; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.956; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.958; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (2)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.966; org.apache.solr.core.CoreContainer; Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368611865504/collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.967; org.apache.solr.cloud.ZkController; Check for collection zkNode:collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.968; org.apache.solr.cloud.ZkController; Collection zkNode exists
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.968; org.apache.solr.cloud.ZkController; Load collection config from:/collections/collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.970; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368611865504/collection1/'
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.972; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368611865504/collection1/lib/README' to classloader
[junit4:junit4] 1> INFO - 2013-05-15 09:57:46.973; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368611865504/collection1/lib/classes/' to classloader
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.025; org.apache.solr.core.SolrConfig; Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.086; org.apache.solr.core.SolrConfig; Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.188; org.apache.solr.schema.IndexSchema; Reading Solr Schema from schema.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.195; org.apache.solr.schema.IndexSchema; [collection1] Schema name=test
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.811; org.apache.solr.schema.IndexSchema; default search field in schema is text
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.816; org.apache.solr.schema.IndexSchema; unique key field: id
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.818; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.824; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.878; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.879; org.apache.solr.core.SolrCore; [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1368611865504/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty1/
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.879; org.apache.solr.core.JmxMonitoredMap; JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@49df819
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.880; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.881; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.881; org.apache.solr.core.SolrCore; New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty1/index/
[junit4:junit4] 1> WARN - 2013-05-15 09:57:47.881; org.apache.solr.core.SolrCore; [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.882; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty1/index
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.886; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@13657737 lockFactory=org.apache.lucene.store.NativeFSLockFactory@413f04a),segFN=segments_1,generation=1}
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.886; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.890; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.890; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.891; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.892; org.apache.solr.core.RequestHandlers; created standard: solr.StandardRequestHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.893; org.apache.solr.core.RequestHandlers; created /get: solr.RealTimeGetHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.893; org.apache.solr.core.RequestHandlers; adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.894; org.apache.solr.core.RequestHandlers; created /replication: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.895; org.apache.solr.core.RequestHandlers; created /update: solr.UpdateRequestHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.896; org.apache.solr.core.RequestHandlers; created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.909; org.apache.solr.handler.loader.XMLLoader; xsltCacheLifetimeSeconds=60
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.916; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@2617b42c main
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.917; org.apache.solr.update.UpdateHandler; Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty1/tlog
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.918; org.apache.solr.update.CommitTracker; Hard AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.918; org.apache.solr.update.CommitTracker; Soft AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.925; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@2617b42c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.928; org.apache.solr.cloud.ZkController; publishing core=collection1 state=down
[junit4:junit4] 1> INFO - 2013-05-15 09:57:47.928; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-05-15 09:57:48.446; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:48.448; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"2",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":null,
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:15017_ofu",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:15017/ofu"}
[junit4:junit4] 1> INFO - 2013-05-15 09:57:48.449; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Create collection collection1 with numShards 2
[junit4:junit4] 1> INFO - 2013-05-15 09:57:48.449; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Assigning new node to shard=shard1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:48.466; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:48.466; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:48.466; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:48.958; org.apache.solr.core.CoreContainer; registering core: collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:48.958; org.apache.solr.cloud.ZkController; Register replica - core:collection1 address:http://127.0.0.1:15017/ofu collection:collection1 shard:shard1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:48.960; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 1> INFO - 2013-05-15 09:57:48.981; org.apache.solr.cloud.ShardLeaderElectionContext; Running the leader process.
[junit4:junit4] 1> INFO - 2013-05-15 09:57:48.997; org.apache.solr.cloud.ShardLeaderElectionContext; Enough replicas found to continue.
[junit4:junit4] 1> INFO - 2013-05-15 09:57:48.998; org.apache.solr.cloud.ShardLeaderElectionContext; I may be the new leader - try and sync
[junit4:junit4] 1> INFO - 2013-05-15 09:57:48.998; org.apache.solr.cloud.SyncStrategy; Sync replicas to http://127.0.0.1:15017/ofu/collection1/
[junit4:junit4] 1> INFO - 2013-05-15 09:57:48.999; org.apache.solr.cloud.SyncStrategy; Sync Success - now sync replicas to me
[junit4:junit4] 1> INFO - 2013-05-15 09:57:48.999; org.apache.solr.cloud.SyncStrategy; http://127.0.0.1:15017/ofu/collection1/ has no replicas
[junit4:junit4] 1> INFO - 2013-05-15 09:57:49.000; org.apache.solr.cloud.ShardLeaderElectionContext; I am the new leader: http://127.0.0.1:15017/ofu/collection1/
[junit4:junit4] 1> INFO - 2013-05-15 09:57:49.000; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:49.974; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:49.995; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:49.995; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:49.995; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.044; org.apache.solr.cloud.ZkController; We are http://127.0.0.1:15017/ofu/collection1/ and leader is http://127.0.0.1:15017/ofu/collection1/
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.044; org.apache.solr.cloud.ZkController; No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:15017/ofu
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.044; org.apache.solr.cloud.ZkController; I am the leader, no recovery necessary
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.045; org.apache.solr.cloud.ZkController; publishing core=collection1 state=active
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.045; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.048; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.050; org.apache.solr.servlet.SolrDispatchFilter; user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.051; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init() done
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.052; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.306; org.eclipse.jetty.server.Server; jetty-8.1.10.v20130312
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.309; org.eclipse.jetty.server.AbstractConnector; Started SocketConnector@127.0.0.1:64346
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.311; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init()
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.311; org.apache.solr.core.SolrResourceLoader; JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.312; org.apache.solr.core.SolrResourceLoader; using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368611870069
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.312; org.apache.solr.core.CoreContainer$Initializer; looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368611870069/solr.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.313; org.apache.solr.core.CoreContainer; New CoreContainer 1372613593
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.314; org.apache.solr.core.CoreContainer; Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368611870069/'
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.315; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368611870069/'
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.435; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting socketTimeout to: 120000
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.436; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting urlScheme to: http://
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.437; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting connTimeout to: 15000
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.437; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxConnectionsPerHost to: 20
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.438; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting corePoolSize to: 0
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.438; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maximumPoolSize to: 2147483647
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.439; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxThreadIdleTime to: 5
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.440; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting sizeOfQueue to: -1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.440; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting fairnessPolicy to: false
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.441; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.457; org.apache.solr.logging.LogWatcher; Registering Log Listener
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.458; org.apache.solr.core.ZkContainer; Zookeeper client=127.0.0.1:20553/solr
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.459; org.apache.solr.cloud.ZkController; zkHost includes chroot
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.460; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.466; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@b186271 name:ZooKeeperConnection Watcher:127.0.0.1:20553 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.467; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.470; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.482; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.485; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@2ff653ab name:ZooKeeperConnection Watcher:127.0.0.1:20553/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.485; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:50.498; org.apache.solr.common.cloud.ZkStateReader; Updating cluster state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.504; org.apache.solr.cloud.ZkController; Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64346_ofu
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.506; org.apache.solr.common.cloud.SolrZkClient; makePath: /live_nodes/127.0.0.1:64346_ofu
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.509; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.510; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.511; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.512; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.512; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.512; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.511; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"2",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:15017_ofu",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:15017/ofu"}
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.513; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.537; org.apache.solr.core.CoreContainer; Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368611870069/collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.537; org.apache.solr.cloud.ZkController; Check for collection zkNode:collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.540; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.540; org.apache.solr.cloud.ZkController; Collection zkNode exists
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.541; org.apache.solr.cloud.ZkController; Load collection config from:/collections/collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.540; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.540; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.543; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368611870069/collection1/'
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.545; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368611870069/collection1/lib/classes/' to classloader
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.546; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368611870069/collection1/lib/README' to classloader
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.547; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.548; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.598; org.apache.solr.core.SolrConfig; Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.659; org.apache.solr.core.SolrConfig; Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.761; org.apache.solr.schema.IndexSchema; Reading Solr Schema from schema.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:51.768; org.apache.solr.schema.IndexSchema; [collection1] Schema name=test
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.377; org.apache.solr.schema.IndexSchema; default search field in schema is text
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.381; org.apache.solr.schema.IndexSchema; unique key field: id
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.383; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.390; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.446; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.447; org.apache.solr.core.SolrCore; [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1368611870069/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty2/
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.448; org.apache.solr.core.JmxMonitoredMap; JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@49df819
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.448; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.449; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty2
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.450; org.apache.solr.core.SolrCore; New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty2/index/
[junit4:junit4] 1> WARN - 2013-05-15 09:57:52.450; org.apache.solr.core.SolrCore; [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.451; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty2/index
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.454; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3d27e959 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4714919),segFN=segments_1,generation=1}
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.455; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.458; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.459; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.460; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.461; org.apache.solr.core.RequestHandlers; created standard: solr.StandardRequestHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.462; org.apache.solr.core.RequestHandlers; created /get: solr.RealTimeGetHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.462; org.apache.solr.core.RequestHandlers; adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.463; org.apache.solr.core.RequestHandlers; created /replication: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.464; org.apache.solr.core.RequestHandlers; created /update: solr.UpdateRequestHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.464; org.apache.solr.core.RequestHandlers; created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.478; org.apache.solr.handler.loader.XMLLoader; xsltCacheLifetimeSeconds=60
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.484; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@2e176c19 main
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.485; org.apache.solr.update.UpdateHandler; Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty2/tlog
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.486; org.apache.solr.update.CommitTracker; Hard AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.486; org.apache.solr.update.CommitTracker; Soft AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.493; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@2e176c19 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.496; org.apache.solr.cloud.ZkController; publishing core=collection1 state=down
[junit4:junit4] 1> INFO - 2013-05-15 09:57:52.496; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.049; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.050; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"3",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":null,
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:64346_ofu",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:64346/ofu"}
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.051; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Collection already exists with numShards=2
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.051; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Assigning new node to shard=shard2
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.083; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.083; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.083; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.083; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.500; org.apache.solr.core.CoreContainer; registering core: collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.500; org.apache.solr.cloud.ZkController; Register replica - core:collection1 address:http://127.0.0.1:64346/ofu collection:collection1 shard:shard2
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.502; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.522; org.apache.solr.cloud.ShardLeaderElectionContext; Running the leader process.
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.536; org.apache.solr.cloud.ShardLeaderElectionContext; Enough replicas found to continue.
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.537; org.apache.solr.cloud.ShardLeaderElectionContext; I may be the new leader - try and sync
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.537; org.apache.solr.cloud.SyncStrategy; Sync replicas to http://127.0.0.1:64346/ofu/collection1/
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.538; org.apache.solr.cloud.SyncStrategy; Sync Success - now sync replicas to me
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.538; org.apache.solr.cloud.SyncStrategy; http://127.0.0.1:64346/ofu/collection1/ has no replicas
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.538; org.apache.solr.cloud.ShardLeaderElectionContext; I am the new leader: http://127.0.0.1:64346/ofu/collection1/
[junit4:junit4] 1> INFO - 2013-05-15 09:57:53.539; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.598; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.617; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.617; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.617; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.617; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.626; org.apache.solr.cloud.ZkController; We are http://127.0.0.1:64346/ofu/collection1/ and leader is http://127.0.0.1:64346/ofu/collection1/
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.626; org.apache.solr.cloud.ZkController; No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:64346/ofu
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.626; org.apache.solr.cloud.ZkController; I am the leader, no recovery necessary
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.627; org.apache.solr.cloud.ZkController; publishing core=collection1 state=active
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.627; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.630; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.632; org.apache.solr.servlet.SolrDispatchFilter; user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.632; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init() done
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.633; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.872; org.eclipse.jetty.server.Server; jetty-8.1.10.v20130312
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.876; org.eclipse.jetty.server.AbstractConnector; Started SocketConnector@127.0.0.1:41442
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.877; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init()
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.878; org.apache.solr.core.SolrResourceLoader; JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.878; org.apache.solr.core.SolrResourceLoader; using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368611874653
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.879; org.apache.solr.core.CoreContainer$Initializer; looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368611874653/solr.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.880; org.apache.solr.core.CoreContainer; New CoreContainer 973379996
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.881; org.apache.solr.core.CoreContainer; Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368611874653/'
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.881; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368611874653/'
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.997; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting socketTimeout to: 120000
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.998; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting urlScheme to: http://
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.999; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting connTimeout to: 15000
[junit4:junit4] 1> INFO - 2013-05-15 09:57:54.999; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxConnectionsPerHost to: 20
[junit4:junit4] 1> INFO - 2013-05-15 09:57:55.000; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting corePoolSize to: 0
[junit4:junit4] 1> INFO - 2013-05-15 09:57:55.000; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maximumPoolSize to: 2147483647
[junit4:junit4] 1> INFO - 2013-05-15 09:57:55.001; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxThreadIdleTime to: 5
[junit4:junit4] 1> INFO - 2013-05-15 09:57:55.002; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting sizeOfQueue to: -1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:55.002; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting fairnessPolicy to: false
[junit4:junit4] 1> INFO - 2013-05-15 09:57:55.003; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 1> INFO - 2013-05-15 09:57:55.021; org.apache.solr.logging.LogWatcher; Registering Log Listener
[junit4:junit4] 1> INFO - 2013-05-15 09:57:55.022; org.apache.solr.core.ZkContainer; Zookeeper client=127.0.0.1:20553/solr
[junit4:junit4] 1> INFO - 2013-05-15 09:57:55.022; org.apache.solr.cloud.ZkController; zkHost includes chroot
[junit4:junit4] 1> INFO - 2013-05-15 09:57:55.024; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:55.028; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@421a1abb name:ZooKeeperConnection Watcher:127.0.0.1:20553 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-05-15 09:57:55.029; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:55.045; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 1> INFO - 2013-05-15 09:57:55.059; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:55.061; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@3d214c59 name:ZooKeeperConnection Watcher:127.0.0.1:20553/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-05-15 09:57:55.062; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:55.069; org.apache.solr.common.cloud.ZkStateReader; Updating cluster state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.074; org.apache.solr.cloud.ZkController; Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41442_ofu
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.080; org.apache.solr.common.cloud.SolrZkClient; makePath: /live_nodes/127.0.0.1:41442_ofu
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.088; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.089; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.090; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.090; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.090; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.090; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.091; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.092; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.094; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.100; org.apache.solr.core.CoreContainer; Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368611874653/collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.100; org.apache.solr.cloud.ZkController; Check for collection zkNode:collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.101; org.apache.solr.cloud.ZkController; Collection zkNode exists
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.102; org.apache.solr.cloud.ZkController; Load collection config from:/collections/collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.104; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368611874653/collection1/'
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.105; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368611874653/collection1/lib/classes/' to classloader
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.106; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368611874653/collection1/lib/README' to classloader
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.127; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.129; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"3",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:64346_ofu",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:64346/ofu"}
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.142; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.142; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.142; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.142; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.142; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.159; org.apache.solr.core.SolrConfig; Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.223; org.apache.solr.core.SolrConfig; Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.325; org.apache.solr.schema.IndexSchema; Reading Solr Schema from schema.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.332; org.apache.solr.schema.IndexSchema; [collection1] Schema name=test
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.917; org.apache.solr.schema.IndexSchema; default search field in schema is text
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.921; org.apache.solr.schema.IndexSchema; unique key field: id
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.924; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.930; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.989; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.989; org.apache.solr.core.SolrCore; [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1368611874653/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty3/
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.990; org.apache.solr.core.JmxMonitoredMap; JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@49df819
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.991; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.992; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty3
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.992; org.apache.solr.core.SolrCore; New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty3/index/
[junit4:junit4] 1> WARN - 2013-05-15 09:57:56.992; org.apache.solr.core.SolrCore; [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.993; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty3/index
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.997; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6120411e lockFactory=org.apache.lucene.store.NativeFSLockFactory@15ecea8a),segFN=segments_1,generation=1}
[junit4:junit4] 1> INFO - 2013-05-15 09:57:56.997; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.001; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.002; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.003; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.004; org.apache.solr.core.RequestHandlers; created standard: solr.StandardRequestHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.005; org.apache.solr.core.RequestHandlers; created /get: solr.RealTimeGetHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.005; org.apache.solr.core.RequestHandlers; adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.006; org.apache.solr.core.RequestHandlers; created /replication: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.006; org.apache.solr.core.RequestHandlers; created /update: solr.UpdateRequestHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.007; org.apache.solr.core.RequestHandlers; created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.021; org.apache.solr.handler.loader.XMLLoader; xsltCacheLifetimeSeconds=60
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.028; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@2648e756 main
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.029; org.apache.solr.update.UpdateHandler; Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty3/tlog
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.030; org.apache.solr.update.CommitTracker; Hard AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.031; org.apache.solr.update.CommitTracker; Soft AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.037; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@2648e756 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.041; org.apache.solr.cloud.ZkController; publishing core=collection1 state=down
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.041; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.649; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.651; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"4",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":null,
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:41442_ofu",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:41442/ofu"}
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.652; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Collection already exists with numShards=2
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.652; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Assigning new node to shard=shard1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.664; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.665; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.665; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.665; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:57.665; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.048; org.apache.solr.core.CoreContainer; registering core: collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.048; org.apache.solr.cloud.ZkController; Register replica - core:collection1 address:http://127.0.0.1:41442/ofu collection:collection1 shard:shard1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.053; org.apache.solr.cloud.ZkController; We are http://127.0.0.1:41442/ofu/collection1/ and leader is http://127.0.0.1:15017/ofu/collection1/
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.053; org.apache.solr.cloud.ZkController; No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41442/ofu
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.054; org.apache.solr.cloud.ZkController; Core needs to recover:collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.054; org.apache.solr.update.DefaultSolrCoreState; Running recovery - first canceling any ongoing recovery
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.061; org.apache.solr.cloud.RecoveryStrategy; Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.062; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.062; org.apache.solr.cloud.RecoveryStrategy; ###### startupVersions=[]
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.063; org.apache.solr.cloud.ZkController; publishing core=collection1 state=recovering
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.063; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.064; org.apache.solr.servlet.SolrDispatchFilter; user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.065; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init() done
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.066; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.066; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.102; org.apache.solr.handler.admin.CoreAdminHandler; Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.310; org.eclipse.jetty.server.Server; jetty-8.1.10.v20130312
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.313; org.eclipse.jetty.server.AbstractConnector; Started SocketConnector@127.0.0.1:10784
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.314; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init()
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.315; org.apache.solr.core.SolrResourceLoader; JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.315; org.apache.solr.core.SolrResourceLoader; using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368611878091
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.316; org.apache.solr.core.CoreContainer$Initializer; looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368611878091/solr.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.317; org.apache.solr.core.CoreContainer; New CoreContainer 1649694453
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.318; org.apache.solr.core.CoreContainer; Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368611878091/'
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.318; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368611878091/'
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.438; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting socketTimeout to: 120000
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.439; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting urlScheme to: http://
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.440; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting connTimeout to: 15000
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.440; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxConnectionsPerHost to: 20
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.441; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting corePoolSize to: 0
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.441; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maximumPoolSize to: 2147483647
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.442; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxThreadIdleTime to: 5
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.442; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting sizeOfQueue to: -1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.443; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting fairnessPolicy to: false
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.444; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.460; org.apache.solr.logging.LogWatcher; Registering Log Listener
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.461; org.apache.solr.core.ZkContainer; Zookeeper client=127.0.0.1:20553/solr
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.461; org.apache.solr.cloud.ZkController; zkHost includes chroot
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.463; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.466; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@1c51fd3d name:ZooKeeperConnection Watcher:127.0.0.1:20553 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.467; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.477; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.489; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.492; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@200a28e2 name:ZooKeeperConnection Watcher:127.0.0.1:20553/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.492; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-05-15 09:57:58.502; org.apache.solr.common.cloud.ZkStateReader; Updating cluster state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.171; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.173; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"4",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"recovering",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:41442_ofu",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:41442/ofu"}
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.214; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.214; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.214; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.214; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.214; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.214; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.509; org.apache.solr.cloud.ZkController; Register node as live in ZooKeeper:/live_nodes/127.0.0.1:10784_ofu
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.511; org.apache.solr.common.cloud.SolrZkClient; makePath: /live_nodes/127.0.0.1:10784_ofu
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.514; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.514; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.514; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.515; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (5)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.515; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (5)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.515; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (5)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.517; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.516; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.517; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (5)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.518; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (5)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.517; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.519; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (5)
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.526; org.apache.solr.core.CoreContainer; Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368611878091/collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.526; org.apache.solr.cloud.ZkController; Check for collection zkNode:collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.527; org.apache.solr.cloud.ZkController; Collection zkNode exists
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.527; org.apache.solr.cloud.ZkController; Load collection config from:/collections/collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.530; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368611878091/collection1/'
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.531; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368611878091/collection1/lib/classes/' to classloader
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.532; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368611878091/collection1/lib/README' to classloader
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.587; org.apache.solr.core.SolrConfig; Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.651; org.apache.solr.core.SolrConfig; Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.753; org.apache.solr.schema.IndexSchema; Reading Solr Schema from schema.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:57:59.760; org.apache.solr.schema.IndexSchema; [collection1] Schema name=test
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.105; org.apache.solr.handler.admin.CoreAdminHandler; Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.106; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/cores params={coreNodeName=4&state=recovering&nodeName=127.0.0.1:41442_ofu&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2004
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.354; org.apache.solr.schema.IndexSchema; default search field in schema is text
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.358; org.apache.solr.schema.IndexSchema; unique key field: id
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.361; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.367; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.415; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.415; org.apache.solr.core.SolrCore; [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1368611878091/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty4/
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.416; org.apache.solr.core.JmxMonitoredMap; JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@49df819
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.417; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.417; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty4
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.418; org.apache.solr.core.SolrCore; New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty4/index/
[junit4:junit4] 1> WARN - 2013-05-15 09:58:00.418; org.apache.solr.core.SolrCore; [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.419; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty4/index
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.423; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30e6857a lockFactory=org.apache.lucene.store.NativeFSLockFactory@243e2c21),segFN=segments_1,generation=1}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.423; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.427; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.427; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.428; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.429; org.apache.solr.core.RequestHandlers; created standard: solr.StandardRequestHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.430; org.apache.solr.core.RequestHandlers; created /get: solr.RealTimeGetHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.430; org.apache.solr.core.RequestHandlers; adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.431; org.apache.solr.core.RequestHandlers; created /replication: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.432; org.apache.solr.core.RequestHandlers; created /update: solr.UpdateRequestHandler
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.433; org.apache.solr.core.RequestHandlers; created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.446; org.apache.solr.handler.loader.XMLLoader; xsltCacheLifetimeSeconds=60
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.452; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@24a45a1f main
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.453; org.apache.solr.update.UpdateHandler; Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty4/tlog
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.454; org.apache.solr.update.CommitTracker; Hard AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.454; org.apache.solr.update.CommitTracker; Soft AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.461; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@24a45a1f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.464; org.apache.solr.cloud.ZkController; publishing core=collection1 state=down
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.465; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.721; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.723; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"5",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":null,
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:10784_ofu",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:10784/ofu"}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.723; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Collection already exists with numShards=2
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.723; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Assigning new node to shard=shard2
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.730; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.730; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.730; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.730; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.730; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:00.730; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.468; org.apache.solr.core.CoreContainer; registering core: collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.468; org.apache.solr.cloud.ZkController; Register replica - core:collection1 address:http://127.0.0.1:10784/ofu collection:collection1 shard:shard2
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.473; org.apache.solr.cloud.ZkController; We are http://127.0.0.1:10784/ofu/collection1/ and leader is http://127.0.0.1:64346/ofu/collection1/
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.474; org.apache.solr.cloud.ZkController; No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:10784/ofu
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.474; org.apache.solr.cloud.ZkController; Core needs to recover:collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.474; org.apache.solr.update.DefaultSolrCoreState; Running recovery - first canceling any ongoing recovery
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.475; org.apache.solr.cloud.RecoveryStrategy; Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.476; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.476; org.apache.solr.cloud.RecoveryStrategy; ###### startupVersions=[]
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.477; org.apache.solr.cloud.ZkController; publishing core=collection1 state=recovering
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.477; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.478; org.apache.solr.servlet.SolrDispatchFilter; user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.479; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init() done
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.480; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.497; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.497; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.500; org.apache.solr.cloud.AbstractFullDistribZkTestBase; Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.500; org.apache.solr.cloud.AbstractDistribZkTestBase; Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.502; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:58:01.512; org.apache.solr.handler.admin.CoreAdminHandler; Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.108; org.apache.solr.cloud.RecoveryStrategy; Attempting to PeerSync from http://127.0.0.1:15017/ofu/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.118; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.137; org.apache.solr.update.PeerSync; PeerSync: core=collection1 url=http://127.0.0.1:41442/ofu START replicas=[http://127.0.0.1:15017/ofu/collection1/] nUpdates=100
[junit4:junit4] 1> WARN - 2013-05-15 09:58:02.148; org.apache.solr.update.PeerSync; no frame of reference to tell of we've missed updates
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.148; org.apache.solr.cloud.RecoveryStrategy; PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.149; org.apache.solr.cloud.RecoveryStrategy; Starting Replication Recovery. core=collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.149; org.apache.solr.cloud.RecoveryStrategy; Begin buffering updates. core=collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.149; org.apache.solr.update.UpdateLog; Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.150; org.apache.solr.cloud.RecoveryStrategy; Attempting to replicate from http://127.0.0.1:15017/ofu/collection1/. core=collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.150; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.236; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.238; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"5",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"recovering",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:10784_ofu",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:10784/ofu"}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.249; org.apache.solr.core.SolrCore; [collection1] webapp=/ofu path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.253; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.256; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@13657737 lockFactory=org.apache.lucene.store.NativeFSLockFactory@413f04a),segFN=segments_1,generation=1}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.257; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.258; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@13657737 lockFactory=org.apache.lucene.store.NativeFSLockFactory@413f04a),segFN=segments_1,generation=1}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@13657737 lockFactory=org.apache.lucene.store.NativeFSLockFactory@413f04a),segFN=segments_2,generation=2}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.259; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 2
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.260; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@50069503 realtime
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.260; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.261; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 13
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.280; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.280; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.280; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.280; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.280; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.280; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.323; org.apache.solr.handler.ReplicationHandler; Commits will be reserved for 10000
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.324; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.347; org.apache.solr.handler.SnapPuller; No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.352; org.apache.solr.handler.ReplicationHandler; Commits will be reserved for 10000
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.353; org.apache.solr.core.SolrCore; [collection1] webapp=/ofu path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=3
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.354; org.apache.solr.handler.SnapPuller; Master's generation: 2
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.355; org.apache.solr.handler.SnapPuller; Slave's generation: 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.355; org.apache.solr.handler.SnapPuller; Starting replication process
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.358; org.apache.solr.core.SolrCore; [collection1] webapp=/ofu path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.360; org.apache.solr.handler.SnapPuller; Number of files in latest index in master: 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.361; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty3/index.20130515125802360
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.362; org.apache.solr.handler.SnapPuller; Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@68587eaa lockFactory=org.apache.lucene.store.NativeFSLockFactory@27bb087) fullCopy=false
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.376; org.apache.solr.core.SolrCore; [collection1] webapp=/ofu path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=5
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.382; org.apache.solr.handler.SnapPuller; Total time taken for download : 0 secs
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.384; org.apache.solr.update.DefaultSolrCoreState; Creating new IndexWriter...
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.385; org.apache.solr.update.DefaultSolrCoreState; Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.444; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6120411e lockFactory=org.apache.lucene.store.NativeFSLockFactory@15ecea8a),segFN=segments_1,generation=1}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6120411e lockFactory=org.apache.lucene.store.NativeFSLockFactory@15ecea8a),segFN=segments_2,generation=2}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.445; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 2
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.445; org.apache.solr.update.DefaultSolrCoreState; New IndexWriter is ready to be used.
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.446; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@20d78458 main
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.448; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@20d78458 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.449; org.apache.solr.core.CachingDirectoryFactory; looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty3/index.20130515125802360 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty3/index.20130515125802360;done=true>>]
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.449; org.apache.solr.core.CachingDirectoryFactory; Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty3/index.20130515125802360
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.449; org.apache.solr.core.CachingDirectoryFactory; Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty3/index.20130515125802360
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.450; org.apache.solr.cloud.RecoveryStrategy; No replay needed. core=collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.450; org.apache.solr.cloud.RecoveryStrategy; Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.450; org.apache.solr.cloud.ZkController; publishing core=collection1 state=active
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.451; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.453; org.apache.solr.cloud.RecoveryStrategy; Finished recovery process. core=collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.504; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.514; org.apache.solr.handler.admin.CoreAdminHandler; Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 1> INFO - 2013-05-15 09:58:02.514; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:10784_ofu&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4] 1> INFO - 2013-05-15 09:58:03.508; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:58:03.787; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:58:03.789; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"4",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:41442_ofu",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:41442/ofu"}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:03.806; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:03.806; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:03.806; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:03.806; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:03.806; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:03.806; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.510; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.516; org.apache.solr.cloud.RecoveryStrategy; Attempting to PeerSync from http://127.0.0.1:64346/ofu/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.516; org.apache.solr.update.PeerSync; PeerSync: core=collection1 url=http://127.0.0.1:10784/ofu START replicas=[http://127.0.0.1:64346/ofu/collection1/] nUpdates=100
[junit4:junit4] 1> WARN - 2013-05-15 09:58:04.517; org.apache.solr.update.PeerSync; no frame of reference to tell of we've missed updates
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.517; org.apache.solr.cloud.RecoveryStrategy; PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.518; org.apache.solr.cloud.RecoveryStrategy; Starting Replication Recovery. core=collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.518; org.apache.solr.cloud.RecoveryStrategy; Begin buffering updates. core=collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.519; org.apache.solr.update.UpdateLog; Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.519; org.apache.solr.cloud.RecoveryStrategy; Attempting to replicate from http://127.0.0.1:64346/ofu/collection1/. core=collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.520; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.530; org.apache.solr.core.SolrCore; [collection1] webapp=/ofu path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.539; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.543; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3d27e959 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4714919),segFN=segments_1,generation=1}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.543; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.544; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3d27e959 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4714919),segFN=segments_1,generation=1}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3d27e959 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4714919),segFN=segments_2,generation=2}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.545; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 2
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.546; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@33c1ceda realtime
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.546; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.547; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.548; org.apache.solr.handler.ReplicationHandler; Commits will be reserved for 10000
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.548; org.apache.solr.handler.SnapPuller; No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.552; org.apache.solr.handler.ReplicationHandler; Commits will be reserved for 10000
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.552; org.apache.solr.core.SolrCore; [collection1] webapp=/ofu path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.553; org.apache.solr.handler.SnapPuller; Master's generation: 2
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.554; org.apache.solr.handler.SnapPuller; Slave's generation: 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.554; org.apache.solr.handler.SnapPuller; Starting replication process
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.558; org.apache.solr.core.SolrCore; [collection1] webapp=/ofu path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.559; org.apache.solr.handler.SnapPuller; Number of files in latest index in master: 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.560; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty4/index.20130515125804560
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.561; org.apache.solr.handler.SnapPuller; Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@44621658 lockFactory=org.apache.lucene.store.NativeFSLockFactory@abbc1ee) fullCopy=false
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.565; org.apache.solr.core.SolrCore; [collection1] webapp=/ofu path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.567; org.apache.solr.handler.SnapPuller; Total time taken for download : 0 secs
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.568; org.apache.solr.update.DefaultSolrCoreState; Creating new IndexWriter...
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.568; org.apache.solr.update.DefaultSolrCoreState; Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.621; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30e6857a lockFactory=org.apache.lucene.store.NativeFSLockFactory@243e2c21),segFN=segments_1,generation=1}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30e6857a lockFactory=org.apache.lucene.store.NativeFSLockFactory@243e2c21),segFN=segments_2,generation=2}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.621; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 2
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.622; org.apache.solr.update.DefaultSolrCoreState; New IndexWriter is ready to be used.
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.622; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@6c0267db main
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.624; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@6c0267db main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.625; org.apache.solr.core.CachingDirectoryFactory; looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty4/index.20130515125804560 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty4/index.20130515125804560;done=true>>]
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.625; org.apache.solr.core.CachingDirectoryFactory; Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty4/index.20130515125804560
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.625; org.apache.solr.core.CachingDirectoryFactory; Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty4/index.20130515125804560
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.626; org.apache.solr.cloud.RecoveryStrategy; No replay needed. core=collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.626; org.apache.solr.cloud.RecoveryStrategy; Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.626; org.apache.solr.cloud.ZkController; publishing core=collection1 state=active
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.627; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-05-15 09:58:04.629; org.apache.solr.cloud.RecoveryStrategy; Finished recovery process. core=collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.313; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.315; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"5",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:10784_ofu",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:10784/ofu"}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.333; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.333; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.333; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.333; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.333; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.333; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.513; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.515; org.apache.solr.cloud.AbstractDistribZkTestBase; Recoveries finished - collection: collection1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.526; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.529; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@18aae7e9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@76e1b416),segFN=segments_1,generation=1}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.530; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.531; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@18aae7e9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@76e1b416),segFN=segments_1,generation=1}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@18aae7e9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@76e1b416),segFN=segments_2,generation=2}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.531; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 2
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.532; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@64633684 main
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.533; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.534; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@64633684 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.535; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.562; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.564; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@13657737 lockFactory=org.apache.lucene.store.NativeFSLockFactory@413f04a),segFN=segments_2,generation=2}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@13657737 lockFactory=org.apache.lucene.store.NativeFSLockFactory@413f04a),segFN=segments_3,generation=3}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.564; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 3
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.565; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@40be5a50 main
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.566; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.567; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@40be5a50 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.581; org.apache.solr.update.SolrCmdDistributor; Distrib commit to:[StdNode: http://127.0.0.1:41442/ofu/collection1/, StdNode: http://127.0.0.1:64346/ofu/collection1/, StdNode: http://127.0.0.1:10784/ofu/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.591; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.592; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3d27e959 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4714919),segFN=segments_2,generation=2}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3d27e959 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4714919),segFN=segments_3,generation=3}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.593; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 3
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.593; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@80352d1 main
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.594; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.595; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@80352d1 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.596; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.598; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.600; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.600; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6120411e lockFactory=org.apache.lucene.store.NativeFSLockFactory@15ecea8a),segFN=segments_2,generation=2}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6120411e lockFactory=org.apache.lucene.store.NativeFSLockFactory@15ecea8a),segFN=segments_3,generation=3}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.601; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 3
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.601; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30e6857a lockFactory=org.apache.lucene.store.NativeFSLockFactory@243e2c21),segFN=segments_2,generation=2}
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30e6857a lockFactory=org.apache.lucene.store.NativeFSLockFactory@243e2c21),segFN=segments_3,generation=3}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.602; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@6814b966 main
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.602; org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 3
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.603; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.604; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@d3fcf1d main
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.604; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@6814b966 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.605; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.605; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.606; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@d3fcf1d main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.607; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.608; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 46
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.610; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.615; org.apache.solr.core.SolrCore; [collection1] webapp=/ofu path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.619; org.apache.solr.core.SolrCore; [collection1] webapp=/ofu path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.623; org.apache.solr.core.SolrCore; [collection1] webapp=/ofu path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:05.627; org.apache.solr.core.SolrCore; [collection1] webapp=/ofu path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.670; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1435093578689806336)} 0 38
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.687; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={update.distrib=FROMLEADER&_version_=-1435093578730700800&update.from=http://127.0.0.1:15017/ofu/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1435093578730700800)} 0 2
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.693; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={update.distrib=FROMLEADER&_version_=-1435093578741186560&update.from=http://127.0.0.1:64346/ofu/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1435093578741186560)} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.695; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1435093578741186560)} 0 10
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.696; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1435093578730700800)} 0 21
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.706; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[0 (1435093578762158080)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.727; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1435093578768449536)]} 0 2
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.728; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[0 (1435093578768449536)]} 0 17
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.732; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[1 (1435093578790469632)]} 0 0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.742; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1435093578795712512)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.743; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[1 (1435093578795712512)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.747; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[2 (1435093578806198272)]} 0 0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.766; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1435093578819829760)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.767; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[2 (1435093578819829760)]} 0 7
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.768; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[2]} 0 16
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.774; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[3 (1435093578833461248)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.783; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[3 (1435093578838704128)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.784; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[3 (1435093578838704128)]} 0 7
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.788; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[4 (1435093578849189888)]} 0 0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.801; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[4 (1435093578857578496)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.802; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[4 (1435093578857578496)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.803; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[4]} 0 10
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.808; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[5 (1435093578870161408)]} 0 0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.818; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[5 (1435093578875404288)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.819; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[5 (1435093578875404288)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.823; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[6 (1435093578885890048)]} 0 0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.836; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[6 (1435093578894278656)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.837; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[6 (1435093578894278656)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.838; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[6]} 0 10
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.843; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[7 (1435093578905812992)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.852; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[7 (1435093578911055872)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.853; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[7 (1435093578911055872)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.857; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[8 (1435093578921541632)]} 0 0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.867; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[8 (1435093578926784512)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.868; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[8 (1435093578926784512)]} 0 7
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.873; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[9 (1435093578937270272)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.885; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[9 (1435093578945658880)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.886; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[9 (1435093578945658880)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.887; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[9]} 0 10
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.891; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[10 (1435093578957193216)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.900; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10 (1435093578961387520)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.901; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[10 (1435093578961387520)]} 0 7
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.905; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[11 (1435093578971873280)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.914; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[11 (1435093578976067584)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.915; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[11 (1435093578976067584)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.919; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[12 (1435093578986553344)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.928; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1435093578990747648)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.929; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[12 (1435093578990747648)]} 0 7
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.934; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[13 (1435093579001233408)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.943; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1435093579006476288)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.944; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[13 (1435093579006476288)]} 0 7
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.950; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[14 (1435093579018010624)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.959; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1435093579023253504)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.960; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[14 (1435093579023253504)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.964; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[15 (1435093579033739264)]} 0 0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.977; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1435093579042127872)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.978; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[15 (1435093579042127872)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.979; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[15]} 0 10
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.984; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[16 (1435093579053662208)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.993; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1435093579058905088)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.994; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[16 (1435093579058905088)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:07.998; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[17 (1435093579069390848)]} 0 0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.009; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1435093579074633728)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.010; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[17 (1435093579074633728)]} 0 7
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.014; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[18 (1435093579086168064)]} 0 0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.024; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1435093579091410944)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.025; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[18 (1435093579091410944)]} 0 7
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.029; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[19 (1435093579101896704)]} 0 0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.037; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1435093579105042432)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.038; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[19 (1435093579105042432)]} 0 7
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.044; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[20 (1435093579116576768)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.053; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1435093579121819648)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.054; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[20 (1435093579121819648)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.059; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[21 (1435093579132305408)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.070; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1435093579140694016)]} 0 0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.072; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[21 (1435093579140694016)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.073; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[21]} 0 10
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.078; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[22 (1435093579152228352)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.087; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1435093579157471232)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.088; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[22 (1435093579157471232)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.092; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[23 (1435093579167956992)]} 0 0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.105; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1435093579176345600)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.106; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[23 (1435093579176345600)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.107; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[23]} 0 10
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.112; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[24 (1435093579187879936)]} 0 0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.124; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1435093579196268544)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.125; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[24 (1435093579196268544)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.126; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[24]} 0 11
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.131; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[25 (1435093579207802880)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.143; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1435093579216191488)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.144; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[25 (1435093579216191488)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.145; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[25]} 0 11
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.150; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[26 (1435093579227725824)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.162; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1435093579236114432)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.163; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[26 (1435093579236114432)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.164; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[26]} 0 10
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.168; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[27 (1435093579247648768)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.177; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1435093579251843072)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.178; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[27 (1435093579251843072)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.182; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[28 (1435093579262328832)]} 0 0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.195; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1435093579270717440)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.196; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[28 (1435093579270717440)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.197; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[28]} 0 10
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.202; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[29 (1435093579282251776)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.214; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1435093579290640384)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.215; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[29 (1435093579290640384)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.216; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[29]} 0 10
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.221; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[30 (1435093579302174720)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.233; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1435093579310563328)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.234; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[30 (1435093579310563328)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.235; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[30]} 0 11
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.240; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[31 (1435093579322097664)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.252; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1435093579330486272)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.253; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[31 (1435093579330486272)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.254; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[31]} 0 10
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.259; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[32 (1435093579342020608)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.268; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1435093579347263488)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.269; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[32 (1435093579347263488)]} 0 7
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.274; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[33 (1435093579357749248)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.286; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1435093579367186432)]} 0 0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.287; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[33 (1435093579367186432)]} 0 5
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.289; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[33]} 0 11
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.293; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[34 (1435093579378720768)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.303; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1435093579383963648)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.304; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[34 (1435093579383963648)]} 0 6
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.308; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[35 (1435093579394449408)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.317; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1435093579399692288)]} 0 0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.319; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[35 (1435093579399692288)]} 0 7
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.323; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[36 (1435093579410178048)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.332; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1435093579414372352)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.333; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[36 (1435093579414372352)]} 0 7
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.337; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={wt=javabin&version=2} {add=[37 (1435093579424858112)]} 0 1
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.348; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:64346/ofu/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1435093579432198144)]} 0 0
[junit4:junit4] 1> INFO - 2013-05-15 09:58:08.349; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/ofu path=/update params={distrib.from=http://127.0.0.1:15017/ofu/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[37 (1435093579432198144)]} 0 5
[junit4:jun
[...truncated too long message...]
1> INFO - 2013-05-15 10:01:21.960; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> INFO - 2013-05-15 10:01:21.985; org.apache.solr.cloud.ChaosMonkey; monkey: stop shard! 10784
[junit4:junit4] 1> INFO - 2013-05-15 10:01:21.985; org.apache.solr.core.CoreContainer; Shutting down CoreContainer instance=1649694453
[junit4:junit4] 1> INFO - 2013-05-15 10:01:22.988; org.apache.solr.cloud.ZkController; publishing core=collection1 state=down
[junit4:junit4] 1> INFO - 2013-05-15 10:01:22.988; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-05-15 10:01:23.016; org.apache.solr.core.SolrCore; [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@5c6908a1
[junit4:junit4] 1> INFO - 2013-05-15 10:01:23.018; org.apache.solr.update.DirectUpdateHandler2; closing DirectUpdateHandler2{commits=4,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=602,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 1> INFO - 2013-05-15 10:01:23.019; org.apache.solr.update.SolrCoreState; Closing SolrCoreState
[junit4:junit4] 1> INFO - 2013-05-15 10:01:23.019; org.apache.solr.update.DefaultSolrCoreState; SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 1> INFO - 2013-05-15 10:01:23.020; org.apache.solr.update.DefaultSolrCoreState; closing IndexWriter with IndexWriterCloser
[junit4:junit4] 1> INFO - 2013-05-15 10:01:23.022; org.apache.solr.core.SolrCore; [collection1] Closing main searcher on request.
[junit4:junit4] 1> INFO - 2013-05-15 10:01:23.023; org.apache.solr.core.CachingDirectoryFactory; Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 1> INFO - 2013-05-15 10:01:23.024; org.apache.solr.core.CachingDirectoryFactory; looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty4;done=false>>]
[junit4:junit4] 1> INFO - 2013-05-15 10:01:23.024; org.apache.solr.core.CachingDirectoryFactory; Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty4
[junit4:junit4] 1> INFO - 2013-05-15 10:01:23.024; org.apache.solr.core.CachingDirectoryFactory; looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty4/index;done=false>>]
[junit4:junit4] 1> INFO - 2013-05-15 10:01:23.025; org.apache.solr.core.CachingDirectoryFactory; Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410/jetty4/index
[junit4:junit4] 1> INFO - 2013-05-15 10:01:23.026; org.apache.solr.cloud.Overseer$ClusterStateUpdater; According to ZK I (id=89693346949693452-127.0.0.1:10784_ofu-n_0000000004) am no longer a leader.
[junit4:junit4] 1> INFO - 2013-05-15 10:01:23.028; org.eclipse.jetty.server.handler.ContextHandler; stopped o.e.j.s.ServletContextHandler{/ofu,null}
[junit4:junit4] 1> INFO - 2013-05-15 10:01:23.098; org.apache.solr.SolrTestCaseJ4; ###Ending testDistribSearch
[junit4:junit4] 1> INFO - 2013-05-15 10:01:23.100; org.apache.solr.cloud.ZkTestServer; connecting to 127.0.0.1:20553 20553
[junit4:junit4] 1> INFO - 2013-05-15 10:01:24.185; org.apache.solr.cloud.ChaosMonkey; monkey: stop shard! 23494
[junit4:junit4] 1> INFO - 2013-05-15 10:01:24.186; org.apache.solr.SolrTestCaseJ4; ###Ending testDistribSearch
[junit4:junit4] 1> INFO - 2013-05-15 10:01:24.187; org.apache.solr.cloud.ZkTestServer; connecting to 127.0.0.1:20553 20553
[junit4:junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeyShardSplitTest-1368611861411
[junit4:junit4] 2> check const of shard1
[junit4:junit4] 2> client0
[junit4:junit4] 2> PROPS:2:{"shard":"shard1","state":"down","core":"collection1","collection":"collection1","node_name":"127.0.0.1:15017_ofu","base_url":"http://127.0.0.1:15017/ofu"}
[junit4:junit4] 2> error contacting client: IOException occured when talking to server at: http://127.0.0.1:15017/ofu/collection1
[junit4:junit4] 2>
[junit4:junit4] 2> client1
[junit4:junit4] 2> PROPS:4:{"shard":"shard1","state":"active","core":"collection1","collection":"collection1","node_name":"127.0.0.1:41442_ofu","base_url":"http://127.0.0.1:41442/ofu"}
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:56
[junit4:junit4] 2>
[junit4:junit4] 2> check const of shard2
[junit4:junit4] 2> client0
[junit4:junit4] 2> PROPS:3:{"shard":"shard2","state":"active","core":"collection1","collection":"collection1","node_name":"127.0.0.1:64346_ofu","base_url":"http://127.0.0.1:64346/ofu","leader":"true"}
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:44
[junit4:junit4] 2>
[junit4:junit4] 2> client1
[junit4:junit4] 2> PROPS:5:{"shard":"shard2","state":"active","core":"collection1","collection":"collection1","node_name":"127.0.0.1:10784_ofu","base_url":"http://127.0.0.1:10784/ofu"}
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:44
[junit4:junit4] 2>
[junit4:junit4] 2> !!!! WARNING: best effort to remove /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1368611861410 FAILED !!!!!
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeyShardSplitTest -Dtests.method=testDistribSearch -Dtests.seed=4E27E2E9750C1754 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr_ME_#Latn -Dtests.timezone=Asia/Bahrain -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] ERROR 223s J0 | ChaosMonkeyShardSplitTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:15017/ofu returned non ok status:500, message:Server Error
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([4E27E2E9750C1754:CFC16CF102537768]:0)
[junit4:junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
[junit4:junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4] > at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
[junit4:junit4] > at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 1> INFO - 2013-05-15 10:01:24.208; org.apache.solr.SolrTestCaseJ4; ###deleteCore
[junit4:junit4] 2> 222810 T182 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4] 1> WARN - 2013-05-15 10:01:24.328; org.apache.solr.common.cloud.ZkStateReader$3; ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 1> INFO - 2013-05-15 10:01:24.328; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> WARN - 2013-05-15 10:01:24.328; org.apache.solr.common.cloud.ZkStateReader$2; ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=sr_ME_#Latn, timezone=Asia/Bahrain
[junit4:junit4] 2> NOTE: FreeBSD 9.0-RELEASE amd64/Oracle Corporation 1.7.0_17 (64-bit)/cpus=16,threads=2,free=160917408,total=362807296
[junit4:junit4] 2> NOTE: All tests run in this JVM: [ClusterStateUpdateTest, TestFuzzyAnalyzedSuggestions, IndexSchemaTest, SpatialFilterTest, QueryResultKeyTest, SynonymTokenizerTest, SOLR749Test, ChaosMonkeyNothingIsSafeTest, TestRemoteStreaming, BasicFunctionalityTest, TestTrie, TestSurroundQueryParser, TestFastWriter, CoreAdminHandlerTest, SliceStateUpdateTest, TestPluginEnable, WordBreakSolrSpellCheckerTest, TestClassNameShortening, TestWriterPerf, CacheHeaderTest, ChaosMonkeyShardSplitTest]
[junit4:junit4] Completed on J0 in 223.09s, 1 test, 1 error <<< FAILURES!
[...truncated 776 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/build.xml:377: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/build.xml:357: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/solr/common-build.xml:437: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/lucene/common-build.xml:1240: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-java7/lucene/common-build.xml:884: There were test failures: 295 suites, 1231 tests, 1 error, 27 ignored (8 assumptions)
Total time: 49 minutes 17 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure