You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2013/07/04 12:05:32 UTC
[JENKINS] Lucene-Solr-Tests-4.3-Java6 - Build # 127 - Still Failing
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.3-Java6/127/
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeyShardSplitTest.testDistribSearch
Error Message:
Server at http://127.0.0.1:64499/nkv returned non ok status:500, message:Server Error
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:64499/nkv returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([2F1C2C510E1FC614:AEFAA2497940A628]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:372)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:228)
at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:679)
Build Log:
[...truncated 8445 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeyShardSplitTest
[junit4:junit4] 1> INFO - 2013-07-04 09:33:06.549; org.apache.solr.BaseDistributedSearchTestCase; Setting hostContext system property: /nkv/
[junit4:junit4] 1> INFO - 2013-07-04 09:33:06.557; org.apache.solr.SolrTestCaseJ4; ###Starting testDistribSearch
[junit4:junit4] 1> INFO - 2013-07-04 09:33:06.570; org.apache.solr.cloud.ZkTestServer; STARTING ZK TEST SERVER
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:06.581; org.apache.solr.cloud.ZkTestServer$ZKServerMain; Starting server
[junit4:junit4] 1> INFO - 2013-07-04 09:33:06.782; org.apache.solr.cloud.ZkTestServer; start zk server on port:57778
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.085; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.190; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@6084fa6a name:ZooKeeperConnection Watcher:127.0.0.1:57778 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.191; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.195; org.apache.solr.common.cloud.SolrZkClient; makePath: /solr
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.279; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.282; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@4ca0187c name:ZooKeeperConnection Watcher:127.0.0.1:57778/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.282; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.295; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.303; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/collection1/shards
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.309; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/control_collection
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.313; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/control_collection/shards
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.319; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.343; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.358; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.359; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/schema.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.465; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.466; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.471; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.472; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/protwords.txt
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.478; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.479; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/currency.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.484; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.485; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.490; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.491; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.497; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.497; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.503; org.apache.solr.cloud.AbstractZkTestCase; put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 1> INFO - 2013-07-04 09:33:07.503; org.apache.solr.common.cloud.SolrZkClient; makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.332; org.eclipse.jetty.server.Server; jetty-8.1.8.v20121106
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.647; org.eclipse.jetty.server.AbstractConnector; Started SocketConnector@127.0.0.1:58441
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.698; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init()
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.699; org.apache.solr.core.SolrResourceLoader; JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.700; org.apache.solr.core.SolrResourceLoader; using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1372930387510
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.700; org.apache.solr.core.CoreContainer$Initializer; looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1372930387510/solr.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.701; org.apache.solr.core.CoreContainer; New CoreContainer 1738236461
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.701; org.apache.solr.core.CoreContainer; Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1372930387510/'
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.702; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1372930387510/'
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.922; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting socketTimeout to: 120000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.923; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting urlScheme to: http://
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.923; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting connTimeout to: 15000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.924; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxConnectionsPerHost to: 20
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.924; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting corePoolSize to: 0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.924; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maximumPoolSize to: 2147483647
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.925; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxThreadIdleTime to: 5
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.925; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting sizeOfQueue to: -1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.926; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting fairnessPolicy to: false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:08.994; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.574; org.apache.solr.core.CoreContainer; Registering Log Listener
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.576; org.apache.solr.core.CoreContainer; Zookeeper client=127.0.0.1:57778/solr
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.597; org.apache.solr.cloud.ZkController; zkHost includes chroot
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.598; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.602; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@45d45d18 name:ZooKeeperConnection Watcher:127.0.0.1:57778 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.603; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.610; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.625; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.628; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@4a5e88f7 name:ZooKeeperConnection Watcher:127.0.0.1:57778/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.628; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.640; org.apache.solr.common.cloud.SolrZkClient; makePath: /overseer/queue
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.656; org.apache.solr.common.cloud.SolrZkClient; makePath: /overseer/collection-queue-work
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.675; org.apache.solr.common.cloud.SolrZkClient; makePath: /live_nodes
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.678; org.apache.solr.cloud.ZkController; Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58441_nkv
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.685; org.apache.solr.common.cloud.SolrZkClient; makePath: /live_nodes/127.0.0.1:58441_nkv
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.690; org.apache.solr.common.cloud.SolrZkClient; makePath: /overseer_elect/election
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.719; org.apache.solr.common.cloud.SolrZkClient; makePath: /overseer_elect/leader
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.723; org.apache.solr.cloud.Overseer; Overseer (id=89976365832536067-127.0.0.1:58441_nkv-n_0000000000) starting
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.753; org.apache.solr.common.cloud.SolrZkClient; makePath: /overseer/queue-work
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.774; org.apache.solr.cloud.OverseerCollectionProcessor; Process current queue of collection creations
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.775; org.apache.solr.common.cloud.SolrZkClient; makePath: /clusterstate.json
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.778; org.apache.solr.common.cloud.SolrZkClient; makePath: /aliases.json
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.781; org.apache.solr.common.cloud.ZkStateReader; Updating cluster state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.816; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Starting to work on the main queue
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.866; org.apache.solr.core.CoreContainer; Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1372930387510/collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.866; org.apache.solr.cloud.ZkController; Check for collection zkNode:control_collection
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.868; org.apache.solr.cloud.ZkController; Collection zkNode exists
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.868; org.apache.solr.cloud.ZkController; Load collection config from:/collections/control_collection
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.871; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1372930387510/collection1/'
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.873; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1372930387510/collection1/lib/README' to classloader
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.874; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1372930387510/collection1/lib/classes/' to classloader
[junit4:junit4] 1> INFO - 2013-07-04 09:33:09.950; org.apache.solr.core.SolrConfig; Using Lucene MatchVersion: LUCENE_43
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.014; org.apache.solr.core.SolrConfig; Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.116; org.apache.solr.schema.IndexSchema; Reading Solr Schema from schema.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.130; org.apache.solr.schema.IndexSchema; [collection1] Schema name=test
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.667; org.apache.solr.schema.IndexSchema; default search field in schema is text
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.670; org.apache.solr.schema.IndexSchema; unique key field: id
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.673; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.678; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.725; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.726; org.apache.solr.core.SolrCore; [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1372930387510/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/control/data/
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.726; org.apache.solr.core.JmxMonitoredMap; JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f315415
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.727; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.728; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/control/data
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.728; org.apache.solr.core.SolrCore; New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/control/data/index/
[junit4:junit4] 1> WARN - 2013-07-04 09:33:10.729; org.apache.solr.core.SolrCore; [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.729; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/control/data/index
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.734; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2ad1e832 lockFactory=org.apache.lucene.store.NativeFSLockFactory@30084a74),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.734; org.apache.solr.core.SolrDeletionPolicy; newest commit = 1[segments_1]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.830; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.830; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.831; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.832; org.apache.solr.core.RequestHandlers; created standard: solr.StandardRequestHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.833; org.apache.solr.core.RequestHandlers; created /get: solr.RealTimeGetHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.833; org.apache.solr.core.RequestHandlers; adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.834; org.apache.solr.core.RequestHandlers; created /replication: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.834; org.apache.solr.core.RequestHandlers; created /update: solr.UpdateRequestHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.835; org.apache.solr.core.RequestHandlers; created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.847; org.apache.solr.handler.loader.XMLLoader; xsltCacheLifetimeSeconds=60
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.857; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@5c07612a main
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.857; org.apache.solr.update.UpdateHandler; Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/control/data/tlog
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.859; org.apache.solr.update.CommitTracker; Hard AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.859; org.apache.solr.update.CommitTracker; Soft AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.863; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@5c07612a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.865; org.apache.solr.cloud.ZkController; publishing core=collection1 state=down
[junit4:junit4] 1> INFO - 2013-07-04 09:33:10.865; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-07-04 09:33:11.329; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:11.330; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=1 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":null,
[junit4:junit4] 1> "numShards":"1",
[junit4:junit4] 1> "shard":null,
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:58441_nkv",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:58441/nkv"}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:11.331; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Create collection control_collection with numShards 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:11.372; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Assigning new node to shard=shard1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:11.403; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:11.896; org.apache.solr.core.CoreContainer; registering core: collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:11.896; org.apache.solr.cloud.ZkController; Register replica - core:collection1 address:http://127.0.0.1:58441/nkv collection:control_collection shard:shard1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:11.922; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:11.947; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 1> INFO - 2013-07-04 09:33:11.970; org.apache.solr.cloud.ShardLeaderElectionContext; Running the leader process.
[junit4:junit4] 1> INFO - 2013-07-04 09:33:11.981; org.apache.solr.cloud.ShardLeaderElectionContext; Enough replicas found to continue.
[junit4:junit4] 1> INFO - 2013-07-04 09:33:11.981; org.apache.solr.cloud.ShardLeaderElectionContext; I may be the new leader - try and sync
[junit4:junit4] 1> INFO - 2013-07-04 09:33:11.982; org.apache.solr.cloud.SyncStrategy; Sync replicas to http://127.0.0.1:58441/nkv/collection1/
[junit4:junit4] 1> INFO - 2013-07-04 09:33:11.982; org.apache.solr.cloud.SyncStrategy; Sync Success - now sync replicas to me
[junit4:junit4] 1> INFO - 2013-07-04 09:33:11.982; org.apache.solr.cloud.SyncStrategy; http://127.0.0.1:58441/nkv/collection1/ has no replicas
[junit4:junit4] 1> INFO - 2013-07-04 09:33:11.983; org.apache.solr.cloud.ShardLeaderElectionContext; I am the new leader: http://127.0.0.1:58441/nkv/collection1/
[junit4:junit4] 1> INFO - 2013-07-04 09:33:11.983; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:12.910; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:12.934; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:12.970; org.apache.solr.cloud.ZkController; We are http://127.0.0.1:58441/nkv/collection1/ and leader is http://127.0.0.1:58441/nkv/collection1/
[junit4:junit4] 1> INFO - 2013-07-04 09:33:12.970; org.apache.solr.cloud.ZkController; No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:58441/nkv
[junit4:junit4] 1> INFO - 2013-07-04 09:33:12.971; org.apache.solr.cloud.ZkController; I am the leader, no recovery necessary
[junit4:junit4] 1> INFO - 2013-07-04 09:33:12.971; org.apache.solr.cloud.ZkController; publishing core=collection1 state=active
[junit4:junit4] 1> INFO - 2013-07-04 09:33:12.972; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-07-04 09:33:12.975; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:12.982; org.apache.solr.servlet.SolrDispatchFilter; user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:12.982; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init() done
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.079; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.107; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.114; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.117; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@2e6b611e name:ZooKeeperConnection Watcher:127.0.0.1:57778/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.118; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.120; org.apache.solr.common.cloud.ZkStateReader; Updating cluster state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.127; org.apache.solr.cloud.ChaosMonkey; monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.419; org.eclipse.jetty.server.Server; jetty-8.1.8.v20121106
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.422; org.eclipse.jetty.server.AbstractConnector; Started SocketConnector@127.0.0.1:64499
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.423; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init()
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.424; org.apache.solr.core.SolrResourceLoader; JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.424; org.apache.solr.core.SolrResourceLoader; using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1372930393127
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.425; org.apache.solr.core.CoreContainer$Initializer; looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1372930393127/solr.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.426; org.apache.solr.core.CoreContainer; New CoreContainer 1755797876
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.426; org.apache.solr.core.CoreContainer; Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1372930393127/'
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.427; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1372930393127/'
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.639; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting socketTimeout to: 120000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.640; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting urlScheme to: http://
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.641; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting connTimeout to: 15000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.641; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxConnectionsPerHost to: 20
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.642; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting corePoolSize to: 0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.643; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maximumPoolSize to: 2147483647
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.643; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxThreadIdleTime to: 5
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.644; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting sizeOfQueue to: -1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.644; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting fairnessPolicy to: false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.645; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.665; org.apache.solr.core.CoreContainer; Registering Log Listener
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.666; org.apache.solr.core.CoreContainer; Zookeeper client=127.0.0.1:57778/solr
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.667; org.apache.solr.cloud.ZkController; zkHost includes chroot
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.668; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.680; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@48b915d name:ZooKeeperConnection Watcher:127.0.0.1:57778 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.681; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.684; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.698; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.701; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@1dd1702e name:ZooKeeperConnection Watcher:127.0.0.1:57778/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.701; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:13.714; org.apache.solr.common.cloud.ZkStateReader; Updating cluster state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.451; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.453; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=1 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"127.0.0.1:58441_nkv_collection1",
[junit4:junit4] 1> "numShards":"1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:58441_nkv",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:58441/nkv"}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.472; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.472; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.472; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.721; org.apache.solr.cloud.ZkController; Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64499_nkv
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.727; org.apache.solr.common.cloud.SolrZkClient; makePath: /live_nodes/127.0.0.1:64499_nkv
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.731; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.732; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.733; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (2)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.733; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.735; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (2)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.735; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (2)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.749; org.apache.solr.core.CoreContainer; Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1372930393127/collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.749; org.apache.solr.cloud.ZkController; Check for collection zkNode:collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.751; org.apache.solr.cloud.ZkController; Collection zkNode exists
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.751; org.apache.solr.cloud.ZkController; Load collection config from:/collections/collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.754; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1372930393127/collection1/'
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.756; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1372930393127/collection1/lib/classes/' to classloader
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.757; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1372930393127/collection1/lib/README' to classloader
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.822; org.apache.solr.core.SolrConfig; Using Lucene MatchVersion: LUCENE_43
[junit4:junit4] 1> INFO - 2013-07-04 09:33:14.898; org.apache.solr.core.SolrConfig; Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.001; org.apache.solr.schema.IndexSchema; Reading Solr Schema from schema.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.021; org.apache.solr.schema.IndexSchema; [collection1] Schema name=test
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.681; org.apache.solr.schema.IndexSchema; default search field in schema is text
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.685; org.apache.solr.schema.IndexSchema; unique key field: id
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.687; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.694; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.749; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.749; org.apache.solr.core.SolrCore; [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1372930393127/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty1/
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.750; org.apache.solr.core.JmxMonitoredMap; JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f315415
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.751; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.752; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.752; org.apache.solr.core.SolrCore; New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty1/index/
[junit4:junit4] 1> WARN - 2013-07-04 09:33:15.753; org.apache.solr.core.SolrCore; [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.753; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty1/index
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.757; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30d46b95 lockFactory=org.apache.lucene.store.NativeFSLockFactory@8bfaf9f),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.757; org.apache.solr.core.SolrDeletionPolicy; newest commit = 1[segments_1]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.761; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.762; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.762; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.763; org.apache.solr.core.RequestHandlers; created standard: solr.StandardRequestHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.764; org.apache.solr.core.RequestHandlers; created /get: solr.RealTimeGetHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.765; org.apache.solr.core.RequestHandlers; adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.765; org.apache.solr.core.RequestHandlers; created /replication: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.766; org.apache.solr.core.RequestHandlers; created /update: solr.UpdateRequestHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.767; org.apache.solr.core.RequestHandlers; created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.776; org.apache.solr.handler.loader.XMLLoader; xsltCacheLifetimeSeconds=60
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.784; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@147834eb main
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.785; org.apache.solr.update.UpdateHandler; Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty1/tlog
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.786; org.apache.solr.update.CommitTracker; Hard AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.786; org.apache.solr.update.CommitTracker; Soft AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.795; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@147834eb main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.799; org.apache.solr.cloud.ZkController; publishing core=collection1 state=down
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.799; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.981; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.983; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":null,
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":null,
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:64499_nkv",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:64499/nkv"}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.983; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Create collection collection1 with numShards 2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.984; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Assigning new node to shard=shard1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.997; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.997; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:15.997; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:16.803; org.apache.solr.core.CoreContainer; registering core: collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:16.803; org.apache.solr.cloud.ZkController; Register replica - core:collection1 address:http://127.0.0.1:64499/nkv collection:collection1 shard:shard1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:16.805; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 1> INFO - 2013-07-04 09:33:16.833; org.apache.solr.cloud.ShardLeaderElectionContext; Running the leader process.
[junit4:junit4] 1> INFO - 2013-07-04 09:33:16.838; org.apache.solr.cloud.ShardLeaderElectionContext; Enough replicas found to continue.
[junit4:junit4] 1> INFO - 2013-07-04 09:33:16.839; org.apache.solr.cloud.ShardLeaderElectionContext; I may be the new leader - try and sync
[junit4:junit4] 1> INFO - 2013-07-04 09:33:16.839; org.apache.solr.cloud.SyncStrategy; Sync replicas to http://127.0.0.1:64499/nkv/collection1/
[junit4:junit4] 1> INFO - 2013-07-04 09:33:16.840; org.apache.solr.cloud.SyncStrategy; Sync Success - now sync replicas to me
[junit4:junit4] 1> INFO - 2013-07-04 09:33:16.840; org.apache.solr.cloud.SyncStrategy; http://127.0.0.1:64499/nkv/collection1/ has no replicas
[junit4:junit4] 1> INFO - 2013-07-04 09:33:16.840; org.apache.solr.cloud.ShardLeaderElectionContext; I am the new leader: http://127.0.0.1:64499/nkv/collection1/
[junit4:junit4] 1> INFO - 2013-07-04 09:33:16.841; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.505; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.527; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.527; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.527; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.565; org.apache.solr.cloud.ZkController; We are http://127.0.0.1:64499/nkv/collection1/ and leader is http://127.0.0.1:64499/nkv/collection1/
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.565; org.apache.solr.cloud.ZkController; No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:64499/nkv
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.565; org.apache.solr.cloud.ZkController; I am the leader, no recovery necessary
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.566; org.apache.solr.cloud.ZkController; publishing core=collection1 state=active
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.566; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.570; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.571; org.apache.solr.servlet.SolrDispatchFilter; user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.572; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init() done
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.573; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.894; org.eclipse.jetty.server.Server; jetty-8.1.8.v20121106
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.897; org.eclipse.jetty.server.AbstractConnector; Started SocketConnector@127.0.0.1:59519
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.898; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init()
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.899; org.apache.solr.core.SolrResourceLoader; JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.899; org.apache.solr.core.SolrResourceLoader; using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1372930397593
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.900; org.apache.solr.core.CoreContainer$Initializer; looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1372930397593/solr.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.901; org.apache.solr.core.CoreContainer; New CoreContainer 1034285299
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.901; org.apache.solr.core.CoreContainer; Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1372930397593/'
[junit4:junit4] 1> INFO - 2013-07-04 09:33:17.902; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1372930397593/'
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.104; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting socketTimeout to: 120000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.105; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting urlScheme to: http://
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.105; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting connTimeout to: 15000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.106; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxConnectionsPerHost to: 20
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.107; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting corePoolSize to: 0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.107; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maximumPoolSize to: 2147483647
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.108; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxThreadIdleTime to: 5
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.108; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting sizeOfQueue to: -1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.109; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting fairnessPolicy to: false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.110; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.130; org.apache.solr.core.CoreContainer; Registering Log Listener
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.131; org.apache.solr.core.CoreContainer; Zookeeper client=127.0.0.1:57778/solr
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.132; org.apache.solr.cloud.ZkController; zkHost includes chroot
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.133; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.136; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@502a5a5e name:ZooKeeperConnection Watcher:127.0.0.1:57778 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.137; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.140; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.153; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.156; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@3669ae9f name:ZooKeeperConnection Watcher:127.0.0.1:57778/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.157; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:18.170; org.apache.solr.common.cloud.ZkStateReader; Updating cluster state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.037; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.039; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"127.0.0.1:64499_nkv_collection1",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:64499_nkv",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:64499/nkv"}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.053; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.053; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.053; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.053; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.175; org.apache.solr.cloud.ZkController; Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59519_nkv
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.178; org.apache.solr.common.cloud.SolrZkClient; makePath: /live_nodes/127.0.0.1:59519_nkv
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.228; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.228; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.228; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (3)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.229; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (3)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.229; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.229; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.230; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (3)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.230; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (3)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.236; org.apache.solr.core.CoreContainer; Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1372930397593/collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.237; org.apache.solr.cloud.ZkController; Check for collection zkNode:collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.237; org.apache.solr.cloud.ZkController; Collection zkNode exists
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.238; org.apache.solr.cloud.ZkController; Load collection config from:/collections/collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.240; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1372930397593/collection1/'
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.241; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1372930397593/collection1/lib/classes/' to classloader
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.241; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1372930397593/collection1/lib/README' to classloader
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.278; org.apache.solr.core.SolrConfig; Using Lucene MatchVersion: LUCENE_43
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.320; org.apache.solr.core.SolrConfig; Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.422; org.apache.solr.schema.IndexSchema; Reading Solr Schema from schema.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.433; org.apache.solr.schema.IndexSchema; [collection1] Schema name=test
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.920; org.apache.solr.schema.IndexSchema; default search field in schema is text
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.923; org.apache.solr.schema.IndexSchema; unique key field: id
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.924; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.929; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.973; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.974; org.apache.solr.core.SolrCore; [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1372930397593/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty2/
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.974; org.apache.solr.core.JmxMonitoredMap; JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f315415
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.975; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.976; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.976; org.apache.solr.core.SolrCore; New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty2/index/
[junit4:junit4] 1> WARN - 2013-07-04 09:33:19.977; org.apache.solr.core.SolrCore; [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.977; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty2/index
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.980; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5c64a871 lockFactory=org.apache.lucene.store.NativeFSLockFactory@115c6cb),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.981; org.apache.solr.core.SolrDeletionPolicy; newest commit = 1[segments_1]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.983; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.984; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.984; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.985; org.apache.solr.core.RequestHandlers; created standard: solr.StandardRequestHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.986; org.apache.solr.core.RequestHandlers; created /get: solr.RealTimeGetHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.986; org.apache.solr.core.RequestHandlers; adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.986; org.apache.solr.core.RequestHandlers; created /replication: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.987; org.apache.solr.core.RequestHandlers; created /update: solr.UpdateRequestHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.988; org.apache.solr.core.RequestHandlers; created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 1> INFO - 2013-07-04 09:33:19.998; org.apache.solr.handler.loader.XMLLoader; xsltCacheLifetimeSeconds=60
[junit4:junit4] 1> INFO - 2013-07-04 09:33:20.006; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@165f5a4 main
[junit4:junit4] 1> INFO - 2013-07-04 09:33:20.006; org.apache.solr.update.UpdateHandler; Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty2/tlog
[junit4:junit4] 1> INFO - 2013-07-04 09:33:20.008; org.apache.solr.update.CommitTracker; Hard AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-07-04 09:33:20.008; org.apache.solr.update.CommitTracker; Soft AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-07-04 09:33:20.015; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@165f5a4 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:20.018; org.apache.solr.cloud.ZkController; publishing core=collection1 state=down
[junit4:junit4] 1> INFO - 2013-07-04 09:33:20.018; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-07-04 09:33:20.560; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:20.561; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":null,
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":null,
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:59519_nkv",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:59519/nkv"}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:20.562; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Collection already exists with numShards=2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:20.562; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Assigning new node to shard=shard2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:20.576; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:20.576; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:20.576; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:20.576; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:21.047; org.apache.solr.core.CoreContainer; registering core: collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:21.047; org.apache.solr.cloud.ZkController; Register replica - core:collection1 address:http://127.0.0.1:59519/nkv collection:collection1 shard:shard2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:21.048; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 1> INFO - 2013-07-04 09:33:21.066; org.apache.solr.cloud.ShardLeaderElectionContext; Running the leader process.
[junit4:junit4] 1> INFO - 2013-07-04 09:33:21.071; org.apache.solr.cloud.ShardLeaderElectionContext; Enough replicas found to continue.
[junit4:junit4] 1> INFO - 2013-07-04 09:33:21.071; org.apache.solr.cloud.ShardLeaderElectionContext; I may be the new leader - try and sync
[junit4:junit4] 1> INFO - 2013-07-04 09:33:21.072; org.apache.solr.cloud.SyncStrategy; Sync replicas to http://127.0.0.1:59519/nkv/collection1/
[junit4:junit4] 1> INFO - 2013-07-04 09:33:21.072; org.apache.solr.cloud.SyncStrategy; Sync Success - now sync replicas to me
[junit4:junit4] 1> INFO - 2013-07-04 09:33:21.072; org.apache.solr.cloud.SyncStrategy; http://127.0.0.1:59519/nkv/collection1/ has no replicas
[junit4:junit4] 1> INFO - 2013-07-04 09:33:21.073; org.apache.solr.cloud.ShardLeaderElectionContext; I am the new leader: http://127.0.0.1:59519/nkv/collection1/
[junit4:junit4] 1> INFO - 2013-07-04 09:33:21.073; org.apache.solr.common.cloud.SolrZkClient; makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.084; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.104; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.104; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.104; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.104; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.108; org.apache.solr.cloud.ZkController; We are http://127.0.0.1:59519/nkv/collection1/ and leader is http://127.0.0.1:59519/nkv/collection1/
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.108; org.apache.solr.cloud.ZkController; No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59519/nkv
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.108; org.apache.solr.cloud.ZkController; I am the leader, no recovery necessary
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.109; org.apache.solr.cloud.ZkController; publishing core=collection1 state=active
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.109; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.114; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.121; org.apache.solr.servlet.SolrDispatchFilter; user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.122; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init() done
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.122; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.395; org.eclipse.jetty.server.Server; jetty-8.1.8.v20121106
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.398; org.eclipse.jetty.server.AbstractConnector; Started SocketConnector@127.0.0.1:52395
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.399; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init()
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.400; org.apache.solr.core.SolrResourceLoader; JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.400; org.apache.solr.core.SolrResourceLoader; using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1372930402141
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.401; org.apache.solr.core.CoreContainer$Initializer; looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1372930402141/solr.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.402; org.apache.solr.core.CoreContainer; New CoreContainer 1365920495
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.403; org.apache.solr.core.CoreContainer; Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1372930402141/'
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.403; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1372930402141/'
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.576; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting socketTimeout to: 120000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.577; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting urlScheme to: http://
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.578; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting connTimeout to: 15000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.578; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxConnectionsPerHost to: 20
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.579; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting corePoolSize to: 0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.580; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maximumPoolSize to: 2147483647
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.580; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxThreadIdleTime to: 5
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.581; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting sizeOfQueue to: -1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.581; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting fairnessPolicy to: false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.582; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.601; org.apache.solr.core.CoreContainer; Registering Log Listener
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.601; org.apache.solr.core.CoreContainer; Zookeeper client=127.0.0.1:57778/solr
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.602; org.apache.solr.cloud.ZkController; zkHost includes chroot
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.603; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.607; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@7ae77ca4 name:ZooKeeperConnection Watcher:127.0.0.1:57778 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.607; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.610; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.624; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.626; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@775978e7 name:ZooKeeperConnection Watcher:127.0.0.1:57778/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.627; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:22.640; org.apache.solr.common.cloud.ZkStateReader; Updating cluster state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.625; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.627; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"127.0.0.1:59519_nkv_collection1",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:59519_nkv",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:59519/nkv"}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.655; org.apache.solr.cloud.ZkController; Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52395_nkv
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.657; org.apache.solr.common.cloud.SolrZkClient; makePath: /live_nodes/127.0.0.1:52395_nkv
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.672; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.672; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.673; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.673; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.674; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.674; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.677; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.677; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.687; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.687; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.688; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.688; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.689; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.691; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.694; org.apache.solr.core.CoreContainer; Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1372930402141/collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.694; org.apache.solr.cloud.ZkController; Check for collection zkNode:collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.696; org.apache.solr.cloud.ZkController; Collection zkNode exists
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.696; org.apache.solr.cloud.ZkController; Load collection config from:/collections/collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.698; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1372930402141/collection1/'
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.700; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1372930402141/collection1/lib/README' to classloader
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.701; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1372930402141/collection1/lib/classes/' to classloader
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.759; org.apache.solr.core.SolrConfig; Using Lucene MatchVersion: LUCENE_43
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.822; org.apache.solr.core.SolrConfig; Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.924; org.apache.solr.schema.IndexSchema; Reading Solr Schema from schema.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:23.939; org.apache.solr.schema.IndexSchema; [collection1] Schema name=test
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.508; org.apache.solr.schema.IndexSchema; default search field in schema is text
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.511; org.apache.solr.schema.IndexSchema; unique key field: id
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.513; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.518; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.560; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.561; org.apache.solr.core.SolrCore; [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1372930402141/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty3/
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.561; org.apache.solr.core.JmxMonitoredMap; JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f315415
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.562; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.562; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty3
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.562; org.apache.solr.core.SolrCore; New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty3/index/
[junit4:junit4] 1> WARN - 2013-07-04 09:33:24.563; org.apache.solr.core.SolrCore; [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.563; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty3/index
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.566; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@72de4f4b lockFactory=org.apache.lucene.store.NativeFSLockFactory@61202afe),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.566; org.apache.solr.core.SolrDeletionPolicy; newest commit = 1[segments_1]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.569; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.570; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.570; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.571; org.apache.solr.core.RequestHandlers; created standard: solr.StandardRequestHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.572; org.apache.solr.core.RequestHandlers; created /get: solr.RealTimeGetHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.572; org.apache.solr.core.RequestHandlers; adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.572; org.apache.solr.core.RequestHandlers; created /replication: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.573; org.apache.solr.core.RequestHandlers; created /update: solr.UpdateRequestHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.573; org.apache.solr.core.RequestHandlers; created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.580; org.apache.solr.handler.loader.XMLLoader; xsltCacheLifetimeSeconds=60
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.585; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@70d5fcfe main
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.585; org.apache.solr.update.UpdateHandler; Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty3/tlog
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.586; org.apache.solr.update.CommitTracker; Hard AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.586; org.apache.solr.update.CommitTracker; Soft AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.591; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@70d5fcfe main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.593; org.apache.solr.cloud.ZkController; publishing core=collection1 state=down
[junit4:junit4] 1> INFO - 2013-07-04 09:33:24.593; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.190; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.192; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":null,
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":null,
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:52395_nkv",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:52395/nkv"}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.192; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Collection already exists with numShards=2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.192; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Assigning new node to shard=shard1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.205; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.206; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.205; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.205; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.205; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.598; org.apache.solr.core.CoreContainer; registering core: collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.598; org.apache.solr.cloud.ZkController; Register replica - core:collection1 address:http://127.0.0.1:52395/nkv collection:collection1 shard:shard1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.603; org.apache.solr.cloud.ZkController; We are http://127.0.0.1:52395/nkv/collection1/ and leader is http://127.0.0.1:64499/nkv/collection1/
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.604; org.apache.solr.cloud.ZkController; No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52395/nkv
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.604; org.apache.solr.cloud.ZkController; Core needs to recover:collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.604; org.apache.solr.update.DefaultSolrCoreState; Running recovery - first canceling any ongoing recovery
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.612; org.apache.solr.cloud.RecoveryStrategy; Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.613; org.apache.solr.cloud.RecoveryStrategy; ###### startupVersions=[]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.613; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.614; org.apache.solr.cloud.ZkController; publishing core=collection1 state=recovering
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.614; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.615; org.apache.solr.servlet.SolrDispatchFilter; user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.616; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init() done
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.616; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.617; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.994; org.eclipse.jetty.server.Server; jetty-8.1.8.v20121106
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.998; org.eclipse.jetty.server.AbstractConnector; Started SocketConnector@127.0.0.1:60566
[junit4:junit4] 1> INFO - 2013-07-04 09:33:25.999; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init()
[junit4:junit4] 1> INFO - 2013-07-04 09:33:26.000; org.apache.solr.core.SolrResourceLoader; JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:26.000; org.apache.solr.core.SolrResourceLoader; using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1372930405640
[junit4:junit4] 1> INFO - 2013-07-04 09:33:26.001; org.apache.solr.core.CoreContainer$Initializer; looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1372930405640/solr.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:26.002; org.apache.solr.core.CoreContainer; New CoreContainer 242639238
[junit4:junit4] 1> INFO - 2013-07-04 09:33:26.002; org.apache.solr.core.CoreContainer; Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1372930405640/'
[junit4:junit4] 1> INFO - 2013-07-04 09:33:26.003; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1372930405640/'
[junit4:junit4] 1> INFO - 2013-07-04 09:33:26.713; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:26.714; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"127.0.0.1:52395_nkv_collection1",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"recovering",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:52395_nkv",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:52395/nkv"}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:26.759; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:26.760; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:26.760; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:26.760; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:26.759; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.243; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting socketTimeout to: 120000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.244; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting urlScheme to: http://
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.245; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting connTimeout to: 15000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.247; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxConnectionsPerHost to: 20
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.248; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting corePoolSize to: 0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.249; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maximumPoolSize to: 2147483647
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.250; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting maxThreadIdleTime to: 5
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.251; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting sizeOfQueue to: -1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.253; org.apache.solr.handler.component.HttpShardHandlerFactory; Setting fairnessPolicy to: false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.254; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.276; org.apache.solr.core.CoreContainer; Registering Log Listener
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.277; org.apache.solr.core.CoreContainer; Zookeeper client=127.0.0.1:57778/solr
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.278; org.apache.solr.cloud.ZkController; zkHost includes chroot
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.280; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.286; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@3be74fe9 name:ZooKeeperConnection Watcher:127.0.0.1:57778 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.287; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.291; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.305; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.307; org.apache.solr.common.cloud.ConnectionManager; Watcher org.apache.solr.common.cloud.ConnectionManager@18145132 name:ZooKeeperConnection Watcher:127.0.0.1:57778/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.308; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.320; org.apache.solr.common.cloud.ZkStateReader; Updating cluster state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.433; org.apache.solr.handler.admin.CoreAdminHandler; Going to wait for coreNodeName: 127.0.0.1:52395_nkv_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.434; org.apache.solr.handler.admin.CoreAdminHandler; Waited coreNodeName: 127.0.0.1:52395_nkv_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 0 seconds.
[junit4:junit4] 1> INFO - 2013-07-04 09:33:27.435; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:52395_nkv_collection1&state=recovering&nodeName=127.0.0.1:52395_nkv&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=8
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.326; org.apache.solr.cloud.ZkController; Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60566_nkv
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.354; org.apache.solr.common.cloud.SolrZkClient; makePath: /live_nodes/127.0.0.1:60566_nkv
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.358; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.359; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.358; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.360; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.359; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.359; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.359; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.363; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.363; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.364; org.apache.solr.common.cloud.ZkStateReader$3; Updating live nodes... (5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.364; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.374; org.apache.solr.core.CoreContainer; Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1372930405640/collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.375; org.apache.solr.cloud.ZkController; Check for collection zkNode:collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.377; org.apache.solr.cloud.ZkController; Collection zkNode exists
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.377; org.apache.solr.cloud.ZkController; Load collection config from:/collections/collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.380; org.apache.solr.core.SolrResourceLoader; new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1372930405640/collection1/'
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.382; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1372930405640/collection1/lib/classes/' to classloader
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.383; org.apache.solr.core.SolrResourceLoader; Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1372930405640/collection1/lib/README' to classloader
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.476; org.apache.solr.core.SolrConfig; Using Lucene MatchVersion: LUCENE_43
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.552; org.apache.solr.core.SolrConfig; Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.654; org.apache.solr.schema.IndexSchema; Reading Solr Schema from schema.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:28.667; org.apache.solr.schema.IndexSchema; [collection1] Schema name=test
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.277; org.apache.solr.schema.IndexSchema; default search field in schema is text
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.282; org.apache.solr.schema.IndexSchema; unique key field: id
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.285; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.291; org.apache.solr.schema.FileExchangeRateProvider; Reloading exchange rates from file currency.xml
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.345; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.346; org.apache.solr.core.SolrCore; [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1372930405640/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty4/
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.347; org.apache.solr.core.JmxMonitoredMap; JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f315415
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.347; org.apache.solr.core.SolrCore; org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.348; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty4
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.349; org.apache.solr.core.SolrCore; New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty4/index/
[junit4:junit4] 1> WARN - 2013-07-04 09:33:29.349; org.apache.solr.core.SolrCore; [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.350; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty4/index
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.354; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@57f897a7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@62766e80),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.355; org.apache.solr.core.SolrDeletionPolicy; newest commit = 1[segments_1]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.359; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.359; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.360; org.apache.solr.update.processor.UpdateRequestProcessorChain; inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.361; org.apache.solr.core.RequestHandlers; created standard: solr.StandardRequestHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.362; org.apache.solr.core.RequestHandlers; created /get: solr.RealTimeGetHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.363; org.apache.solr.core.RequestHandlers; adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.363; org.apache.solr.core.RequestHandlers; created /replication: solr.ReplicationHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.364; org.apache.solr.core.RequestHandlers; created /update: solr.UpdateRequestHandler
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.365; org.apache.solr.core.RequestHandlers; created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.375; org.apache.solr.handler.loader.XMLLoader; xsltCacheLifetimeSeconds=60
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.383; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@5bda657 main
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.383; org.apache.solr.update.UpdateHandler; Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty4/tlog
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.385; org.apache.solr.update.CommitTracker; Hard AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.385; org.apache.solr.update.CommitTracker; Soft AutoCommit: disabled
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.392; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@5bda657 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.395; org.apache.solr.cloud.ZkController; publishing core=collection1 state=down
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.395; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.647; org.apache.solr.cloud.RecoveryStrategy; Attempting to PeerSync from http://127.0.0.1:64499/nkv/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.659; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.692; org.apache.solr.update.PeerSync; PeerSync: core=collection1 url=http://127.0.0.1:52395/nkv START replicas=[http://127.0.0.1:64499/nkv/collection1/] nUpdates=100
[junit4:junit4] 1> WARN - 2013-07-04 09:33:29.707; org.apache.solr.update.PeerSync; no frame of reference to tell of we've missed updates
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.707; org.apache.solr.cloud.RecoveryStrategy; PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.708; org.apache.solr.cloud.RecoveryStrategy; Starting Replication Recovery. core=collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.708; org.apache.solr.cloud.RecoveryStrategy; Begin buffering updates. core=collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.708; org.apache.solr.update.UpdateLog; Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.709; org.apache.solr.cloud.RecoveryStrategy; Attempting to replicate from http://127.0.0.1:64499/nkv/collection1/. core=collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.709; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.773; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.775; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":null,
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":null,
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:60566_nkv",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:60566/nkv"}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.775; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Collection already exists with numShards=2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.775; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Assigning new node to shard=shard2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.782; org.apache.solr.core.SolrCore; [collection1] webapp=/nkv path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=7
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.794; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.795; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.795; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.795; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.795; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.794; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.811; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.815; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30d46b95 lockFactory=org.apache.lucene.store.NativeFSLockFactory@8bfaf9f),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.815; org.apache.solr.core.SolrDeletionPolicy; newest commit = 1[segments_1]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.816; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30d46b95 lockFactory=org.apache.lucene.store.NativeFSLockFactory@8bfaf9f),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30d46b95 lockFactory=org.apache.lucene.store.NativeFSLockFactory@8bfaf9f),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.817; org.apache.solr.core.SolrDeletionPolicy; newest commit = 2[segments_2]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.818; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@3c079e4c realtime
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.818; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.819; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 44
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.820; org.apache.solr.handler.ReplicationHandler; Commits will be reserved for 10000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.821; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.846; org.apache.solr.handler.SnapPuller; No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.917; org.apache.solr.handler.ReplicationHandler; Commits will be reserved for 10000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.918; org.apache.solr.core.SolrCore; [collection1] webapp=/nkv path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=69
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.920; org.apache.solr.handler.SnapPuller; Master's generation: 2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.920; org.apache.solr.handler.SnapPuller; Slave's generation: 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.920; org.apache.solr.handler.SnapPuller; Starting replication process
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.924; org.apache.solr.core.SolrCore; [collection1] webapp=/nkv path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.926; org.apache.solr.handler.SnapPuller; Number of files in latest index in master: 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.927; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty3/index.20130704043329926
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.927; org.apache.solr.handler.SnapPuller; Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@7dd66aa9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@384e57ba) fullCopy=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.943; org.apache.solr.core.SolrCore; [collection1] webapp=/nkv path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.960; org.apache.solr.handler.SnapPuller; Total time taken for download : 0 secs
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.967; org.apache.solr.update.DefaultSolrCoreState; Creating new IndexWriter...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.968; org.apache.solr.update.DefaultSolrCoreState; Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.970; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@72de4f4b lockFactory=org.apache.lucene.store.NativeFSLockFactory@61202afe),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@72de4f4b lockFactory=org.apache.lucene.store.NativeFSLockFactory@61202afe),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.971; org.apache.solr.core.SolrDeletionPolicy; newest commit = 2[segments_2]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.971; org.apache.solr.update.DefaultSolrCoreState; New IndexWriter is ready to be used.
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.972; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@84301c9 main
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.974; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@84301c9 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.975; org.apache.solr.core.CachingDirectoryFactory; looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty3/index.20130704043329926 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty3/index.20130704043329926;done=true>>]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.975; org.apache.solr.core.CachingDirectoryFactory; Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty3/index.20130704043329926
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.975; org.apache.solr.core.CachingDirectoryFactory; Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty3/index.20130704043329926
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.976; org.apache.solr.cloud.RecoveryStrategy; No replay needed. core=collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.976; org.apache.solr.cloud.RecoveryStrategy; Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.977; org.apache.solr.cloud.ZkController; publishing core=collection1 state=active
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.977; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-07-04 09:33:29.984; org.apache.solr.cloud.RecoveryStrategy; Finished recovery process. core=collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.410; org.apache.solr.core.CoreContainer; registering core: collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.410; org.apache.solr.cloud.ZkController; Register replica - core:collection1 address:http://127.0.0.1:60566/nkv collection:collection1 shard:shard2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.441; org.apache.solr.cloud.ZkController; We are http://127.0.0.1:60566/nkv/collection1/ and leader is http://127.0.0.1:59519/nkv/collection1/
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.441; org.apache.solr.cloud.ZkController; No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60566/nkv
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.441; org.apache.solr.cloud.ZkController; Core needs to recover:collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.442; org.apache.solr.update.DefaultSolrCoreState; Running recovery - first canceling any ongoing recovery
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.442; org.apache.solr.cloud.RecoveryStrategy; Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.443; org.apache.solr.cloud.RecoveryStrategy; ###### startupVersions=[]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.444; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.444; org.apache.solr.cloud.ZkController; publishing core=collection1 state=recovering
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.445; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.446; org.apache.solr.servlet.SolrDispatchFilter; user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.446; org.apache.solr.servlet.SolrDispatchFilter; SolrDispatchFilter.init() done
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.447; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.447; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.467; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.468; org.apache.solr.handler.admin.CoreAdminHandler; Going to wait for coreNodeName: 127.0.0.1:60566_nkv_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.469; org.apache.solr.cloud.AbstractFullDistribZkTestBase; Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.470; org.apache.solr.cloud.AbstractDistribZkTestBase; Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 1> INFO - 2013-07-04 09:33:30.471; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:31.303; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:31.305; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"127.0.0.1:52395_nkv_collection1",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:52395_nkv",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:52395/nkv"}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:31.336; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"127.0.0.1:60566_nkv_collection1",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"recovering",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:60566_nkv",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:60566/nkv"}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:31.351; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:31.351; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:31.351; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:31.351; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:31.351; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:31.351; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:31.470; org.apache.solr.handler.admin.CoreAdminHandler; Waited coreNodeName: 127.0.0.1:60566_nkv_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 1> INFO - 2013-07-04 09:33:31.470; org.apache.solr.servlet.SolrDispatchFilter; [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:60566_nkv_collection1&state=recovering&nodeName=127.0.0.1:60566_nkv&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4] 1> INFO - 2013-07-04 09:33:31.475; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:32.478; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.473; org.apache.solr.cloud.RecoveryStrategy; Attempting to PeerSync from http://127.0.0.1:59519/nkv/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.473; org.apache.solr.update.PeerSync; PeerSync: core=collection1 url=http://127.0.0.1:60566/nkv START replicas=[http://127.0.0.1:59519/nkv/collection1/] nUpdates=100
[junit4:junit4] 1> WARN - 2013-07-04 09:33:33.474; org.apache.solr.update.PeerSync; no frame of reference to tell of we've missed updates
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.475; org.apache.solr.cloud.RecoveryStrategy; PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.475; org.apache.solr.cloud.RecoveryStrategy; Starting Replication Recovery. core=collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.475; org.apache.solr.cloud.RecoveryStrategy; Begin buffering updates. core=collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.476; org.apache.solr.update.UpdateLog; Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.476; org.apache.solr.cloud.RecoveryStrategy; Attempting to replicate from http://127.0.0.1:59519/nkv/collection1/. core=collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.477; org.apache.solr.client.solrj.impl.HttpClientUtil; Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.482; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.487; org.apache.solr.core.SolrCore; [collection1] webapp=/nkv path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.491; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.494; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5c64a871 lockFactory=org.apache.lucene.store.NativeFSLockFactory@115c6cb),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.495; org.apache.solr.core.SolrDeletionPolicy; newest commit = 1[segments_1]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.496; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5c64a871 lockFactory=org.apache.lucene.store.NativeFSLockFactory@115c6cb),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5c64a871 lockFactory=org.apache.lucene.store.NativeFSLockFactory@115c6cb),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.496; org.apache.solr.core.SolrDeletionPolicy; newest commit = 2[segments_2]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.497; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@2210a0e realtime
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.498; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.498; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.499; org.apache.solr.handler.ReplicationHandler; Commits will be reserved for 10000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.500; org.apache.solr.handler.SnapPuller; No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.503; org.apache.solr.handler.ReplicationHandler; Commits will be reserved for 10000
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.503; org.apache.solr.core.SolrCore; [collection1] webapp=/nkv path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.505; org.apache.solr.handler.SnapPuller; Master's generation: 2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.505; org.apache.solr.handler.SnapPuller; Slave's generation: 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.505; org.apache.solr.handler.SnapPuller; Starting replication process
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.509; org.apache.solr.core.SolrCore; [collection1] webapp=/nkv path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.510; org.apache.solr.handler.SnapPuller; Number of files in latest index in master: 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.511; org.apache.solr.core.CachingDirectoryFactory; return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty4/index.20130704043333510
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.511; org.apache.solr.handler.SnapPuller; Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@6d8b6b0e lockFactory=org.apache.lucene.store.NativeFSLockFactory@20e2e869) fullCopy=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.515; org.apache.solr.core.SolrCore; [collection1] webapp=/nkv path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.516; org.apache.solr.handler.SnapPuller; Total time taken for download : 0 secs
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.518; org.apache.solr.update.DefaultSolrCoreState; Creating new IndexWriter...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.518; org.apache.solr.update.DefaultSolrCoreState; Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.521; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@57f897a7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@62766e80),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@57f897a7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@62766e80),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.521; org.apache.solr.core.SolrDeletionPolicy; newest commit = 2[segments_2]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.522; org.apache.solr.update.DefaultSolrCoreState; New IndexWriter is ready to be used.
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.522; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@332af08b main
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.524; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@332af08b main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.525; org.apache.solr.core.CachingDirectoryFactory; looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty4/index.20130704043333510 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty4/index.20130704043333510;done=true>>]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.525; org.apache.solr.core.CachingDirectoryFactory; Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty4/index.20130704043333510
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.525; org.apache.solr.core.CachingDirectoryFactory; Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty4/index.20130704043333510
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.526; org.apache.solr.cloud.RecoveryStrategy; No replay needed. core=collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.526; org.apache.solr.cloud.RecoveryStrategy; Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.527; org.apache.solr.cloud.ZkController; publishing core=collection1 state=active
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.527; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-07-04 09:33:33.529; org.apache.solr.cloud.RecoveryStrategy; Finished recovery process. core=collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.371; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.373; org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update state numShards=2 message={
[junit4:junit4] 1> "operation":"state",
[junit4:junit4] 1> "core_node_name":"127.0.0.1:60566_nkv_collection1",
[junit4:junit4] 1> "numShards":"2",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "shard_state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard_range":null,
[junit4:junit4] 1> "node_name":"127.0.0.1:60566_nkv",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:60566/nkv"}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.406; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.407; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.407; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.407; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.407; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.407; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.484; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.486; org.apache.solr.cloud.AbstractDistribZkTestBase; Recoveries finished - collection: collection1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.495; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.499; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2ad1e832 lockFactory=org.apache.lucene.store.NativeFSLockFactory@30084a74),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.500; org.apache.solr.core.SolrDeletionPolicy; newest commit = 1[segments_1]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.501; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2ad1e832 lockFactory=org.apache.lucene.store.NativeFSLockFactory@30084a74),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2ad1e832 lockFactory=org.apache.lucene.store.NativeFSLockFactory@30084a74),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.501; org.apache.solr.core.SolrDeletionPolicy; newest commit = 2[segments_2]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.502; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@1fa80a2d main
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.503; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.504; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@1fa80a2d main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.505; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 10
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.531; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.532; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30d46b95 lockFactory=org.apache.lucene.store.NativeFSLockFactory@8bfaf9f),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30d46b95 lockFactory=org.apache.lucene.store.NativeFSLockFactory@8bfaf9f),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.533; org.apache.solr.core.SolrDeletionPolicy; newest commit = 3[segments_3]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.534; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@6668a2cc main
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.534; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.535; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@6668a2cc main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.546; org.apache.solr.update.SolrCmdDistributor; Distrib commit to:[StdNode: http://127.0.0.1:52395/nkv/collection1/, StdNode: http://127.0.0.1:59519/nkv/collection1/, StdNode: http://127.0.0.1:60566/nkv/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.558; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.559; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5c64a871 lockFactory=org.apache.lucene.store.NativeFSLockFactory@115c6cb),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5c64a871 lockFactory=org.apache.lucene.store.NativeFSLockFactory@115c6cb),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.560; org.apache.solr.core.SolrDeletionPolicy; newest commit = 3[segments_3]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.560; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@44594680 main
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.561; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.562; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@44594680 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.563; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.565; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.565; org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.566; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@72de4f4b lockFactory=org.apache.lucene.store.NativeFSLockFactory@61202afe),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@72de4f4b lockFactory=org.apache.lucene.store.NativeFSLockFactory@61202afe),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.567; org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@57f897a7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@62766e80),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4] 1> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@57f897a7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@62766e80),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.567; org.apache.solr.core.SolrDeletionPolicy; newest commit = 3[segments_3]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.567; org.apache.solr.core.SolrDeletionPolicy; newest commit = 3[segments_3]
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.568; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@5dd6c8b9 main
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.569; org.apache.solr.search.SolrIndexSearcher; Opening Searcher@67aac78d main
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.569; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.570; org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.570; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@5dd6c8b9 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.571; org.apache.solr.core.SolrCore; [collection1] Registered new searcher Searcher@67aac78d main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.572; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.572; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.573; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 42
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.575; org.apache.solr.common.cloud.ZkStateReader; Updating cloud state from ZooKeeper...
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.599; org.apache.solr.core.SolrCore; [collection1] webapp=/nkv path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.603; org.apache.solr.core.SolrCore; [collection1] webapp=/nkv path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.606; org.apache.solr.core.SolrCore; [collection1] webapp=/nkv path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:34.609; org.apache.solr.core.SolrCore; [collection1] webapp=/nkv path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.730; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1439621884650979328)} 0 36
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.746; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={update.distrib=FROMLEADER&_version_=-1439621884657270784&update.from=http://127.0.0.1:64499/nkv/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439621884657270784)} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.751; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={update.distrib=FROMLEADER&_version_=-1439621884668805120&update.from=http://127.0.0.1:59519/nkv/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439621884668805120)} 0 2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.752; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1439621884668805120)} 0 7
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.753; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1439621884657270784)} 0 19
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.758; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[0 (1439621884681388032)]} 0 2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.779; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439621884686630912)]} 0 2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.780; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[0 (1439621884686630912)]} 0 19
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.784; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[1 (1439621884709699584)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.792; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439621884713893888)]} 0 0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.793; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[1 (1439621884713893888)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.798; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[2 (1439621884724379648)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.820; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1439621884742205440)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.821; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[2 (1439621884742205440)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.822; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[2]} 0 21
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.826; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[3 (1439621884753739776)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.835; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[3 (1439621884757934080)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.836; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[3 (1439621884757934080)]} 0 7
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.840; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[4 (1439621884768419840)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.849; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[4 (1439621884772614144)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.850; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[4 (1439621884772614144)]} 0 7
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.856; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[5 (1439621884784148480)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.869; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[5 (1439621884793585664)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.870; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[5 (1439621884793585664)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.871; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[5]} 0 11
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.875; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[6 (1439621884805120000)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.885; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[6 (1439621884810362880)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.886; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[6 (1439621884810362880)]} 0 7
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.891; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[7 (1439621884820848640)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.900; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[7 (1439621884826091520)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.901; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[7 (1439621884826091520)]} 0 7
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.905; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[8 (1439621884836577280)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.918; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[8 (1439621884843917312)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.919; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[8 (1439621884843917312)]} 0 7
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.920; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[8]} 0 11
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.924; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[9 (1439621884856500224)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.933; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[9 (1439621884860694528)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.934; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[9 (1439621884860694528)]} 0 7
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.939; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[10 (1439621884871180288)]} 0 2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.951; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10 (1439621884879568896)]} 0 0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.951; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[10 (1439621884879568896)]} 0 5
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.952; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[10]} 0 10
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.957; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[11 (1439621884891103232)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.963; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[11 (1439621884895297536)]} 0 0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.964; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[11 (1439621884895297536)]} 0 4
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.969; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[12 (1439621884902637568)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.978; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1439621884907880448)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.979; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[12 (1439621884907880448)]} 0 7
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.983; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[13 (1439621884918366208)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.995; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1439621884925706240)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.996; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[13 (1439621884925706240)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:36.997; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[13]} 0 11
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.001; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[14 (1439621884937240576)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.010; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1439621884941434880)]} 0 0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.011; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[14 (1439621884941434880)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.015; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[15 (1439621884951920640)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.027; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1439621884959260672)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.028; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[15 (1439621884959260672)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.029; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[15]} 0 11
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.034; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[16 (1439621884970795008)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.046; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1439621884979183616)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.047; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[16 (1439621884979183616)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.048; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[16]} 0 11
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.052; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[17 (1439621884990717952)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.063; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1439621884998057984)]} 0 0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.064; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[17 (1439621884998057984)]} 0 5
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.066; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[17]} 0 11
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.070; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[18 (1439621885009592320)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.082; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1439621885016932352)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.083; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[18 (1439621885016932352)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.083; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[18]} 0 9
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.088; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[19 (1439621885028466688)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.100; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1439621885035806720)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.101; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[19 (1439621885035806720)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.102; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[19]} 0 10
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.106; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[20 (1439621885047341056)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.117; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1439621885052583936)]} 0 2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.118; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[20 (1439621885052583936)]} 0 8
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.124; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[21 (1439621885065166848)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.136; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1439621885073555456)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.137; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[21 (1439621885073555456)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.138; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[21]} 0 10
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.143; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[22 (1439621885085089792)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.152; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1439621885090332672)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.153; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[22 (1439621885090332672)]} 0 7
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.157; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[23 (1439621885100818432)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.170; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1439621885109207040)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.171; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[23 (1439621885109207040)]} 0 7
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.172; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[23]} 0 11
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.177; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[24 (1439621885120741376)]} 0 2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.189; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1439621885129129984)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.191; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[24 (1439621885129129984)]} 0 7
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.193; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[24]} 0 13
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.199; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[25 (1439621885143810048)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.207; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1439621885149052928)]} 0 0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.208; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[25 (1439621885149052928)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.212; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[26 (1439621885158490112)]} 0 0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.225; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1439621885166878720)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.226; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[26 (1439621885166878720)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.227; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[26]} 0 11
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.231; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[27 (1439621885178413056)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.244; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1439621885186801664)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.245; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[27 (1439621885186801664)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.246; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[27]} 0 11
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.250; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[28 (1439621885198336000)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.262; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1439621885205676032)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.263; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[28 (1439621885205676032)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.264; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[28]} 0 11
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.268; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[29 (1439621885217210368)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.278; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1439621885222453248)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.279; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[29 (1439621885222453248)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.283; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[30 (1439621885232939008)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.295; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1439621885240279040)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.295; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[30 (1439621885240279040)]} 0 5
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.296; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[30]} 0 10
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.300; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[31 (1439621885250764800)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.308; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1439621885254959104)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.309; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[31 (1439621885254959104)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.314; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[32 (1439621885265444864)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.323; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1439621885270687744)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.324; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[32 (1439621885270687744)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.329; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[33 (1439621885281173504)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.339; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1439621885286416384)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.340; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[33 (1439621885286416384)]} 0 7
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.344; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[34 (1439621885296902144)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.352; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1439621885301096448)]} 0 0
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.353; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[34 (1439621885301096448)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.358; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[35 (1439621885311582208)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.370; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1439621885319970816)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.371; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[35 (1439621885319970816)]} 0 6
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.373; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[35]} 0 11
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.378; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[36 (1439621885331505152)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.387; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1439621885336748032)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.388; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[36 (1439621885336748032)]} 0 7
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.393; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={wt=javabin&version=2} {add=[37 (1439621885347233792)]} 0 2
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.405; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:59519/nkv/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1439621885355622400)]} 0 1
[junit4:junit4] 1> INFO - 2013-07-04 09:33:37.406; org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=/nkv path=/update params={distrib.from=http://127.0.0.1:64499/nkv/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[37 (1439621885355622400)]} 0 6
[junit4:jun
[...truncated too long message...]
he.solr.cloud.ChaosMonkey; monkey: stop shard! 60566
[junit4:junit4] 1> INFO - 2013-07-04 09:37:11.566; org.apache.solr.core.CoreContainer; Shutting down CoreContainer instance=242639238
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.569; org.apache.solr.cloud.ZkController; publishing core=collection1 state=down
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.569; org.apache.solr.cloud.ZkController; numShards not found on descriptor - reading it from system property
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.578; org.apache.solr.core.SolrCore; [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@2d62bdd8
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.580; org.apache.solr.update.DirectUpdateHandler2; closing DirectUpdateHandler2{commits=4,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=679,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.581; org.apache.solr.update.SolrCoreState; Closing SolrCoreState
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.581; org.apache.solr.update.DefaultSolrCoreState; SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.582; org.apache.solr.update.DefaultSolrCoreState; closing IndexWriter with IndexWriterCloser
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.584; org.apache.solr.core.SolrCore; [collection1] Closing main searcher on request.
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.585; org.apache.solr.core.CachingDirectoryFactory; Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.585; org.apache.solr.core.CachingDirectoryFactory; looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty4/index;done=false>>]
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.585; org.apache.solr.core.CachingDirectoryFactory; Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty4/index
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.586; org.apache.solr.core.CachingDirectoryFactory; looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty4;done=false>>]
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.586; org.apache.solr.core.CachingDirectoryFactory; Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558/jetty4
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.587; org.apache.solr.cloud.Overseer$ClusterStateUpdater; According to ZK I (id=89976365832536076-127.0.0.1:60566_nkv-n_0000000004) am no longer a leader.
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.594; org.eclipse.jetty.server.handler.ContextHandler; stopped o.e.j.s.ServletContextHandler{/nkv,null}
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.669; org.apache.solr.SolrTestCaseJ4; ###Ending testDistribSearch
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.673; org.apache.solr.cloud.ZkTestServer; connecting to 127.0.0.1:57778 57778
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.799; org.apache.solr.cloud.ChaosMonkey; monkey: stop shard! 58441
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.800; org.apache.solr.SolrTestCaseJ4; ###Ending testDistribSearch
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.801; org.apache.solr.cloud.ZkTestServer; connecting to 127.0.0.1:57778 57778
[junit4:junit4] 1> WARN - 2013-07-04 09:37:12.813; org.apache.solr.common.cloud.ZkStateReader$3; ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.813; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 1> WARN - 2013-07-04 09:37:12.813; org.apache.solr.common.cloud.ZkStateReader$2; ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeyShardSplitTest-1372930386559
[junit4:junit4] 2> check const of shard1
[junit4:junit4] 2> client0
[junit4:junit4] 2> PROPS:127.0.0.1:64499_nkv_collection1:{"shard":"shard1","state":"down","core":"collection1","collection":"collection1","node_name":"127.0.0.1:64499_nkv","base_url":"http://127.0.0.1:64499/nkv"}
[junit4:junit4] 2> error contacting client: IOException occured when talking to server at: http://127.0.0.1:64499/nkv/collection1
[junit4:junit4] 2>
[junit4:junit4] 2> client1
[junit4:junit4] 2> PROPS:127.0.0.1:52395_nkv_collection1:{"shard":"shard1","state":"active","core":"collection1","collection":"collection1","node_name":"127.0.0.1:52395_nkv","base_url":"http://127.0.0.1:52395/nkv"}
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:56
[junit4:junit4] 2>
[junit4:junit4] 2> check const of shard2
[junit4:junit4] 2> client0
[junit4:junit4] 2> PROPS:127.0.0.1:59519_nkv_collection1:{"shard":"shard2","state":"active","core":"collection1","collection":"collection1","node_name":"127.0.0.1:59519_nkv","base_url":"http://127.0.0.1:59519/nkv","leader":"true"}
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:44
[junit4:junit4] 2>
[junit4:junit4] 2> client1
[junit4:junit4] 2> PROPS:127.0.0.1:60566_nkv_collection1:{"shard":"shard2","state":"active","core":"collection1","collection":"collection1","node_name":"127.0.0.1:60566_nkv","base_url":"http://127.0.0.1:60566/nkv"}
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:44
[junit4:junit4] 2>
[junit4:junit4] 2> !!!! WARNING: best effort to remove /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1372930386558 FAILED !!!!!
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeyShardSplitTest -Dtests.method=testDistribSearch -Dtests.seed=2F1C2C510E1FC614 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_TN -Dtests.timezone=America/Guayaquil -Dtests.file.encoding=ISO8859-1
[junit4:junit4] ERROR 246s J1 | ChaosMonkeyShardSplitTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:64499/nkv returned non ok status:500, message:Server Error
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([2F1C2C510E1FC614:AEFAA2497940A628]:0)
[junit4:junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:372)
[junit4:junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4] > at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:228)
[junit4:junit4] > at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:679)
[junit4:junit4] 1> INFO - 2013-07-04 09:37:12.966; org.apache.solr.SolrTestCaseJ4; ###deleteCore
[junit4:junit4] 2> 246423 T24 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 1> WARN - 2013-07-04 09:37:13.894; org.apache.solr.common.cloud.ZkStateReader$3; ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 1> INFO - 2013-07-04 09:37:13.894; org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 1> WARN - 2013-07-04 09:37:13.894; org.apache.solr.common.cloud.ZkStateReader$2; ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=ar_TN, timezone=America/Guayaquil
[junit4:junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=2,free=348677856,total=426377216
[junit4:junit4] 2> NOTE: All tests run in this JVM: [DebugComponentTest, CopyFieldTest, SpellCheckCollatorTest, ConvertedLegacyTest, NumericFieldsTest, ChaosMonkeyShardSplitTest]
[junit4:junit4] Completed on J1 in 247.47s, 1 test, 1 error <<< FAILURES!
[...truncated 797 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/build.xml:381: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/build.xml:361: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/solr/common-build.xml:437: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/lucene/common-build.xml:1243: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.3-Java6/lucene/common-build.xml:887: There were test failures: 292 suites, 1209 tests, 1 error, 13 ignored (7 assumptions)
Total time: 52 minutes 29 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure