You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/05/28 06:56:05 UTC
[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 498 -
Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/498/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseParallelGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeyShardSplitTest.testDistribSearch
Error Message:
Server at http://127.0.0.1:52431/df returned non ok status:500, message:Server Error
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:52431/df returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([C345B394745EAB26:42A33D8C0301CB1A]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9396 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeyShardSplitTest
[junit4:junit4] 2> 1006823 T1783 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /df/
[junit4:junit4] 2> 1006838 T1783 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeyShardSplitTest-1369713886003
[junit4:junit4] 2> 1006842 T1783 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 1006845 T1784 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 1006947 T1783 oasc.ZkTestServer.run start zk server on port:52424
[junit4:junit4] 2> 1006950 T1783 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1006959 T1790 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17d526e5 name:ZooKeeperConnection Watcher:127.0.0.1:52424 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1006960 T1783 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1006960 T1783 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 1006980 T1783 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1006988 T1792 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@24426f41 name:ZooKeeperConnection Watcher:127.0.0.1:52424/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1006988 T1783 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1006989 T1783 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 1007004 T1783 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 1007017 T1783 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 1007028 T1783 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 1007041 T1783 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 1007043 T1783 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 1007057 T1783 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 1007058 T1783 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 1007077 T1783 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 1007079 T1783 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 1007089 T1783 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 1007091 T1783 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 1007099 T1783 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 1007101 T1783 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 1007110 T1783 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 1007112 T1783 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 1007122 T1783 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 1007123 T1783 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 1007141 T1783 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 1007142 T1783 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 1007152 T1783 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 1007154 T1783 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 1007939 T1783 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 1007957 T1783 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52427
[junit4:junit4] 2> 1007958 T1783 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 1007959 T1783 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 1007959 T1783 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369713886328
[junit4:junit4] 2> 1007960 T1783 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369713886328/solr.xml
[junit4:junit4] 2> 1007961 T1783 oasc.CoreContainer.<init> New CoreContainer 172202967
[junit4:junit4] 2> 1007962 T1783 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369713886328/'
[junit4:junit4] 2> 1007964 T1783 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369713886328/'
[junit4:junit4] 2> 1008237 T1783 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 1008237 T1783 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 1008238 T1783 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 1008238 T1783 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 1008239 T1783 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 1008239 T1783 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 1008239 T1783 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 1008240 T1783 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 1008240 T1783 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 1008241 T1783 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 1008279 T1783 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 1008280 T1783 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52424/solr
[junit4:junit4] 2> 1008281 T1783 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 1008284 T1783 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1008292 T1803 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7fa49ae7 name:ZooKeeperConnection Watcher:127.0.0.1:52424 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1008294 T1783 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1008301 T1783 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 1008337 T1783 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1008342 T1805 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19338d5f name:ZooKeeperConnection Watcher:127.0.0.1:52424/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1008343 T1783 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1008349 T1783 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 1008362 T1783 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 1008374 T1783 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 1008381 T1783 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52427_df
[junit4:junit4] 2> 1008384 T1783 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52427_df
[junit4:junit4] 2> 1008395 T1783 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 1008421 T1783 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 1008433 T1783 oasc.Overseer.start Overseer (id=89765569234206723-127.0.0.1:52427_df-n_0000000000) starting
[junit4:junit4] 2> 1008444 T1783 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 1008457 T1807 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 1008464 T1783 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 1008479 T1783 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 1008489 T1783 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 1008500 T1806 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 1008513 T1808 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369713886328/collection1
[junit4:junit4] 2> 1008513 T1808 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 1008515 T1808 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 1008516 T1808 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 1008520 T1808 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369713886328/collection1/'
[junit4:junit4] 2> 1008523 T1808 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369713886328/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 1008523 T1808 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369713886328/collection1/lib/README' to classloader
[junit4:junit4] 2> 1008622 T1808 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 1008764 T1808 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 1008768 T1808 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 1008776 T1808 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 1009904 T1808 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 1009912 T1808 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 1009917 T1808 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1009931 T1808 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1010058 T1808 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1010059 T1808 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369713886328/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/control/data/
[junit4:junit4] 2> 1010060 T1808 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ed56fe5
[junit4:junit4] 2> 1010061 T1808 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1010062 T1808 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/control/data
[junit4:junit4] 2> 1010063 T1808 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/control/data/index/
[junit4:junit4] 2> 1010064 T1808 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 1010065 T1808 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/control/data/index
[junit4:junit4] 2> 1010074 T1808 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@56d0ddb8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c24cb1b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1010075 T1808 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1010083 T1808 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 1010085 T1808 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 1010091 T1808 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 1010095 T1808 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 1010096 T1808 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 1010097 T1808 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 1010097 T1808 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 1010098 T1808 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 1010099 T1808 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 1010125 T1808 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 1010140 T1808 oass.SolrIndexSearcher.<init> Opening Searcher@4d9fc7ad main
[junit4:junit4] 2> 1010141 T1808 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/control/data/tlog
[junit4:junit4] 2> 1010144 T1808 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 1010145 T1808 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 1010153 T1809 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4d9fc7ad main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 1010156 T1808 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1010156 T1808 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1011530 T1806 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1011533 T1806 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52427_df",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52427/df"}
[junit4:junit4] 2> 1011533 T1806 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 1011534 T1806 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 1011550 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1012163 T1808 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 1012163 T1808 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52427/df collection:control_collection shard:shard1
[junit4:junit4] 2> 1012166 T1808 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 1012203 T1808 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 1012211 T1808 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 1012211 T1808 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 1012212 T1808 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52427/df/collection1/
[junit4:junit4] 2> 1012212 T1808 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 1012213 T1808 oasc.SyncStrategy.syncToMe http://127.0.0.1:52427/df/collection1/ has no replicas
[junit4:junit4] 2> 1012213 T1808 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52427/df/collection1/
[junit4:junit4] 2> 1012213 T1808 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 1013071 T1806 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1013104 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1013160 T1808 oasc.ZkController.register We are http://127.0.0.1:52427/df/collection1/ and leader is http://127.0.0.1:52427/df/collection1/
[junit4:junit4] 2> 1013161 T1808 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52427/df
[junit4:junit4] 2> 1013161 T1808 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 1013161 T1808 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 1013161 T1808 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1013165 T1808 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1013169 T1783 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 1013169 T1783 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 1013170 T1783 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1013203 T1783 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 1013206 T1783 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1013212 T1812 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bc51f4e name:ZooKeeperConnection Watcher:127.0.0.1:52424/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1013213 T1783 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1013218 T1783 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 1013227 T1783 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 1013839 T1783 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 1013849 T1783 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52431
[junit4:junit4] 2> 1013850 T1783 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 1013851 T1783 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 1013851 T1783 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369713892390
[junit4:junit4] 2> 1013852 T1783 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369713892390/solr.xml
[junit4:junit4] 2> 1013852 T1783 oasc.CoreContainer.<init> New CoreContainer 1521339240
[junit4:junit4] 2> 1013853 T1783 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369713892390/'
[junit4:junit4] 2> 1013854 T1783 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369713892390/'
[junit4:junit4] 2> 1014072 T1783 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 1014073 T1783 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 1014073 T1783 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 1014074 T1783 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 1014075 T1783 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 1014075 T1783 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 1014076 T1783 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 1014076 T1783 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 1014077 T1783 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 1014078 T1783 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 1014106 T1783 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 1014107 T1783 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52424/solr
[junit4:junit4] 2> 1014107 T1783 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 1014113 T1783 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1014118 T1823 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3771eba name:ZooKeeperConnection Watcher:127.0.0.1:52424 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1014119 T1783 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1014127 T1783 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 1014183 T1783 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1014190 T1825 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5508e99f name:ZooKeeperConnection Watcher:127.0.0.1:52424/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1014192 T1783 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1014210 T1783 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 1014628 T1806 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1014630 T1806 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:52427_df_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52427_df",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52427/df"}
[junit4:junit4] 2> 1014644 T1825 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1014645 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1014645 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1015223 T1783 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52431_df
[junit4:junit4] 2> 1015228 T1783 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52431_df
[junit4:junit4] 2> 1015238 T1825 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1015239 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1015241 T1812 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 1015241 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1015250 T1805 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 1015257 T1825 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 1015270 T1826 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369713892390/collection1
[junit4:junit4] 2> 1015271 T1826 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 1015273 T1826 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 1015273 T1826 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 1015277 T1826 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369713892390/collection1/'
[junit4:junit4] 2> 1015280 T1826 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369713892390/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 1015281 T1826 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369713892390/collection1/lib/README' to classloader
[junit4:junit4] 2> 1015368 T1826 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 1015466 T1826 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 1015470 T1826 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 1015477 T1826 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 1016435 T1826 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 1016441 T1826 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 1016444 T1826 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1016462 T1826 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1016549 T1826 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1016550 T1826 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369713892390/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty1/
[junit4:junit4] 2> 1016550 T1826 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ed56fe5
[junit4:junit4] 2> 1016551 T1826 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1016552 T1826 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty1
[junit4:junit4] 2> 1016552 T1826 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty1/index/
[junit4:junit4] 2> 1016552 T1826 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 1016554 T1826 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty1/index
[junit4:junit4] 2> 1016560 T1826 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@32eeca40 lockFactory=org.apache.lucene.store.NativeFSLockFactory@56889e1c),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1016561 T1826 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1016566 T1826 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 1016566 T1826 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 1016568 T1826 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 1016570 T1826 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 1016571 T1826 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 1016572 T1826 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 1016572 T1826 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 1016573 T1826 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 1016574 T1826 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 1016599 T1826 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 1016639 T1826 oass.SolrIndexSearcher.<init> Opening Searcher@ee27ba main
[junit4:junit4] 2> 1016640 T1826 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty1/tlog
[junit4:junit4] 2> 1016642 T1826 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 1016642 T1826 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 1016648 T1827 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ee27ba main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 1016652 T1826 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1016652 T1826 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1017693 T1806 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1017702 T1806 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52431_df",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52431/df"}
[junit4:junit4] 2> 1017703 T1806 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 1017703 T1806 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 1017730 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1017730 T1825 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1017731 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1018660 T1826 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 1018661 T1826 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52431/df collection:collection1 shard:shard1
[junit4:junit4] 2> 1018668 T1826 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 1018703 T1826 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 1018722 T1826 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 1018723 T1826 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 1018723 T1826 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52431/df/collection1/
[junit4:junit4] 2> 1018723 T1826 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 1018724 T1826 oasc.SyncStrategy.syncToMe http://127.0.0.1:52431/df/collection1/ has no replicas
[junit4:junit4] 2> 1018724 T1826 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52431/df/collection1/
[junit4:junit4] 2> 1018724 T1826 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 1019268 T1806 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1019311 T1825 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1019313 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1019314 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1019322 T1826 oasc.ZkController.register We are http://127.0.0.1:52431/df/collection1/ and leader is http://127.0.0.1:52431/df/collection1/
[junit4:junit4] 2> 1019322 T1826 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52431/df
[junit4:junit4] 2> 1019323 T1826 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 1019324 T1826 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 1019325 T1826 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1019334 T1826 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1019349 T1783 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 1019352 T1783 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 1019353 T1783 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1020430 T1783 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 1020472 T1783 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52434
[junit4:junit4] 2> 1020478 T1783 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 1020481 T1783 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 1020483 T1783 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369713898575
[junit4:junit4] 2> 1020484 T1783 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369713898575/solr.xml
[junit4:junit4] 2> 1020485 T1783 oasc.CoreContainer.<init> New CoreContainer 655466152
[junit4:junit4] 2> 1020486 T1783 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369713898575/'
[junit4:junit4] 2> 1020488 T1783 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369713898575/'
[junit4:junit4] 2> 1020821 T1783 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 1020822 T1783 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 1020823 T1783 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 1020823 T1783 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 1020824 T1783 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 1020824 T1783 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 1020825 T1783 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 1020835 T1783 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 1020836 T1783 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 1020836 T1783 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 1020875 T1783 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 1020884 T1783 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52424/solr
[junit4:junit4] 2> 1020885 T1783 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 1020896 T1783 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1020901 T1806 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1020909 T1806 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:52431_df_collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52431_df",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52431/df"}
[junit4:junit4] 2> 1020912 T1839 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70973084 name:ZooKeeperConnection Watcher:127.0.0.1:52424 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1020912 T1783 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1020951 T1783 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 1020954 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1020953 T1825 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1020953 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1021003 T1783 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1021012 T1841 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e936a53 name:ZooKeeperConnection Watcher:127.0.0.1:52424/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1021014 T1783 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1021078 T1783 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 1022121 T1783 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52434_df
[junit4:junit4] 2> 1022128 T1783 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52434_df
[junit4:junit4] 2> 1022142 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1022146 T1825 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1022151 T1812 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 1022152 T1841 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 1022152 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1022157 T1805 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 1022163 T1825 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 1022181 T1842 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369713898575/collection1
[junit4:junit4] 2> 1022182 T1842 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 1022187 T1842 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 1022187 T1842 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 1022196 T1842 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369713898575/collection1/'
[junit4:junit4] 2> 1022197 T1842 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369713898575/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 1022198 T1842 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369713898575/collection1/lib/README' to classloader
[junit4:junit4] 2> 1022299 T1842 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 1022488 T1842 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 1022495 T1842 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 1022508 T1842 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 1023948 T1842 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 1023965 T1842 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 1023969 T1842 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1023982 T1842 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1024246 T1842 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1024248 T1842 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369713898575/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty2/
[junit4:junit4] 2> 1024248 T1842 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ed56fe5
[junit4:junit4] 2> 1024250 T1842 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1024254 T1842 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty2
[junit4:junit4] 2> 1024255 T1842 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty2/index/
[junit4:junit4] 2> 1024256 T1842 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 1024262 T1842 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty2/index
[junit4:junit4] 2> 1024269 T1842 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f255dc5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@d7b5bd4),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1024270 T1842 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1024280 T1842 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 1024281 T1842 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 1024283 T1842 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 1024285 T1842 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 1024287 T1842 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 1024288 T1842 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 1024289 T1842 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 1024290 T1842 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 1024292 T1842 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 1024328 T1842 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 1024378 T1842 oass.SolrIndexSearcher.<init> Opening Searcher@52b3d4a0 main
[junit4:junit4] 2> 1024379 T1842 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty2/tlog
[junit4:junit4] 2> 1024382 T1842 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 1024382 T1842 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 1024394 T1843 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@52b3d4a0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 1024401 T1842 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1024402 T1842 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1025514 T1806 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1025521 T1806 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52434_df",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52434/df"}
[junit4:junit4] 2> 1025522 T1806 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 1025523 T1806 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 1025549 T1841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1025550 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1025551 T1825 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1025552 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1026410 T1842 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 1026411 T1842 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52434/df collection:collection1 shard:shard2
[junit4:junit4] 2> 1026413 T1842 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 1026447 T1842 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 1026457 T1842 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 1026458 T1842 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 1026459 T1842 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52434/df/collection1/
[junit4:junit4] 2> 1026459 T1842 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 1026460 T1842 oasc.SyncStrategy.syncToMe http://127.0.0.1:52434/df/collection1/ has no replicas
[junit4:junit4] 2> 1026460 T1842 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52434/df/collection1/
[junit4:junit4] 2> 1026461 T1842 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 1027079 T1806 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1027107 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1027107 T1841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1027109 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1027109 T1825 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1027154 T1842 oasc.ZkController.register We are http://127.0.0.1:52434/df/collection1/ and leader is http://127.0.0.1:52434/df/collection1/
[junit4:junit4] 2> 1027155 T1842 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52434/df
[junit4:junit4] 2> 1027156 T1842 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 1027156 T1842 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 1027157 T1842 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1027163 T1842 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1027167 T1783 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 1027168 T1783 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 1027169 T1783 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1028199 T1783 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 1028222 T1783 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52437
[junit4:junit4] 2> 1028224 T1783 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 1028225 T1783 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 1028226 T1783 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369713906377
[junit4:junit4] 2> 1028227 T1783 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369713906377/solr.xml
[junit4:junit4] 2> 1028228 T1783 oasc.CoreContainer.<init> New CoreContainer 1890794987
[junit4:junit4] 2> 1028229 T1783 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369713906377/'
[junit4:junit4] 2> 1028231 T1783 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369713906377/'
[junit4:junit4] 2> 1028618 T1783 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 1028619 T1783 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 1028619 T1783 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 1028620 T1783 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 1028621 T1783 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 1028622 T1783 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 1028622 T1783 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 1028623 T1783 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 1028624 T1783 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 1028624 T1783 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 1028652 T1806 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1028657 T1806 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:52434_df_collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52434_df",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52434/df"}
[junit4:junit4] 2> 1028686 T1783 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 1028703 T1783 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52424/solr
[junit4:junit4] 2> 1028704 T1783 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 1028698 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1028698 T1825 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1028697 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1028697 T1841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1028736 T1783 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1028739 T1855 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a030e2a name:ZooKeeperConnection Watcher:127.0.0.1:52424 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1028741 T1783 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1028785 T1783 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 1028826 T1783 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1028841 T1857 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@79c04bb1 name:ZooKeeperConnection Watcher:127.0.0.1:52424/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1028845 T1783 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1028943 T1783 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 1030000 T1783 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52437_df
[junit4:junit4] 2> 1030011 T1783 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52437_df
[junit4:junit4] 2> 1030024 T1825 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1030027 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 1030033 T1812 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 1030033 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1030034 T1857 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 1030036 T1841 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 1030036 T1841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1030046 T1825 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 1030047 T1805 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 1030073 T1858 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369713906377/collection1
[junit4:junit4] 2> 1030073 T1858 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 1030077 T1858 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 1030078 T1858 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 1030083 T1858 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369713906377/collection1/'
[junit4:junit4] 2> 1030086 T1858 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369713906377/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 1030086 T1858 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369713906377/collection1/lib/README' to classloader
[junit4:junit4] 2> 1030207 T1858 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 1030389 T1858 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 1030398 T1858 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 1030405 T1858 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 1031736 T1858 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 1031745 T1858 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 1031751 T1858 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1031772 T1858 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1031913 T1858 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1031915 T1858 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369713906377/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty3/
[junit4:junit4] 2> 1031915 T1858 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ed56fe5
[junit4:junit4] 2> 1031918 T1858 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1031919 T1858 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty3
[junit4:junit4] 2> 1031919 T1858 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty3/index/
[junit4:junit4] 2> 1031920 T1858 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 1031922 T1858 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty3/index
[junit4:junit4] 2> 1031931 T1858 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@14414fe5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@190ba1c3),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1031933 T1858 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1031940 T1858 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 1031941 T1858 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 1031942 T1858 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 1031943 T1858 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 1031945 T1858 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 1031945 T1858 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 1031946 T1858 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 1031947 T1858 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 1031948 T1858 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 1031983 T1858 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 1031999 T1858 oass.SolrIndexSearcher.<init> Opening Searcher@3f971d31 main
[junit4:junit4] 2> 1032000 T1858 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty3/tlog
[junit4:junit4] 2> 1032003 T1858 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 1032004 T1858 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 1032014 T1859 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3f971d31 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 1032020 T1858 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1032020 T1858 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1033266 T1806 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1033268 T1806 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52437_df",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52437/df"}
[junit4:junit4] 2> 1033269 T1806 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 1033269 T1806 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 1033347 T1857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1033348 T1841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1033349 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1033349 T1825 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1033350 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1034028 T1858 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 1034029 T1858 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52437/df collection:collection1 shard:shard1
[junit4:junit4] 2> 1034044 T1858 oasc.ZkController.register We are http://127.0.0.1:52437/df/collection1/ and leader is http://127.0.0.1:52431/df/collection1/
[junit4:junit4] 2> 1034045 T1858 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52437/df
[junit4:junit4] 2> 1034045 T1858 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 1034046 T1858 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C676 name=collection1 org.apache.solr.core.SolrCore@7b3bd807 url=http://127.0.0.1:52437/df/collection1 node=127.0.0.1:52437_df C676_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:52437_df, base_url=http://127.0.0.1:52437/df}
[junit4:junit4] 2> 1034048 T1860 C676 P52437 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 1034052 T1860 C676 P52437 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 1034056 T1860 C676 P52437 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 1034057 T1860 C676 P52437 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1034063 T1858 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1034065 T1860 C676 P52437 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1034069 T1783 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 1034070 T1783 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 1034071 T1783 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1034122 T1820 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:52437_df_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 1034875 T1806 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1034878 T1806 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:52437_df_collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52437_df",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52437/df"}
[junit4:junit4] 2> 1034893 T1857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1034894 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1034894 T1825 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1034896 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1034895 T1841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1034983 T1783 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 1034995 T1783 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52441
[junit4:junit4] 2> 1034996 T1783 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 1034997 T1783 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 1034997 T1783 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369713913270
[junit4:junit4] 2> 1034998 T1783 oasc.CoreContainer$Initializer.initialize looking for solr config file: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369713913270/solr.xml
[junit4:junit4] 2> 1034999 T1783 oasc.CoreContainer.<init> New CoreContainer 1946064216
[junit4:junit4] 2> 1034999 T1783 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369713913270/'
[junit4:junit4] 2> 1035000 T1783 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369713913270/'
[junit4:junit4] 2> 1035124 T1820 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:52437_df_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 1035125 T1820 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:52437_df_collection1&state=recovering&nodeName=127.0.0.1:52437_df&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1003
[junit4:junit4] 2> 1035255 T1783 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 1035256 T1783 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 1035256 T1783 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 1035256 T1783 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 1035257 T1783 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 1035257 T1783 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 1035258 T1783 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 1035258 T1783 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 1035259 T1783 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 1035259 T1783 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 1035292 T1783 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 1035294 T1783 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52424/solr
[junit4:junit4] 2> 1035295 T1783 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 1035297 T1783 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1035304 T1872 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58c9a974 name:ZooKeeperConnection Watcher:127.0.0.1:52424 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1035305 T1783 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1035313 T1783 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 1035350 T1783 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 1035359 T1874 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35ffa460 name:ZooKeeperConnection Watcher:127.0.0.1:52424/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 1035359 T1783 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 1035389 T1783 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 1036402 T1783 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52441_df
[junit4:junit4] 2> 1036404 T1783 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52441_df
[junit4:junit4] 2> 1036414 T1825 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1036416 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 1036426 T1857 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 1036426 T1857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1036426 T1874 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 1036427 T1841 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 1036427 T1841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1036427 T1812 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 1036429 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1036436 T1805 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 1036439 T1825 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 1036454 T1875 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369713913270/collection1
[junit4:junit4] 2> 1036455 T1875 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 1036457 T1875 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 1036457 T1875 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 1036461 T1875 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369713913270/collection1/'
[junit4:junit4] 2> 1036463 T1875 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369713913270/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 1036468 T1875 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369713913270/collection1/lib/README' to classloader
[junit4:junit4] 2> 1036547 T1875 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 1036650 T1875 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 1036655 T1875 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 1036663 T1875 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> ASYNC NEW_CORE C677 name=collection1 org.apache.solr.core.SolrCore@7b3bd807 url=http://127.0.0.1:52437/df/collection1 node=127.0.0.1:52437_df C677_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:52437_df, base_url=http://127.0.0.1:52437/df}
[junit4:junit4] 2> 1037128 T1860 C677 P52437 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:52431/df/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 1037129 T1860 C677 P52437 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52437/df START replicas=[http://127.0.0.1:52431/df/collection1/] nUpdates=100
[junit4:junit4] 2> 1037130 T1860 C677 P52437 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 1037156 T1860 C677 P52437 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 1037158 T1860 C677 P52437 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 1037158 T1860 C677 P52437 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 1037158 T1860 C677 P52437 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 1037160 T1860 C677 P52437 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:52431/df/collection1/. core=collection1
[junit4:junit4] 2> 1037161 T1860 C677 P52437 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C678 name=collection1 org.apache.solr.core.SolrCore@3383994a url=http://127.0.0.1:52431/df/collection1 node=127.0.0.1:52431_df C678_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52431_df, base_url=http://127.0.0.1:52431/df, leader=true}
[junit4:junit4] 2> 1037173 T1817 C678 P52431 oasc.SolrCore.execute [collection1] webapp=/df path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 1037187 T1818 C678 P52431 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 1037232 T1818 C678 P52431 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@32eeca40 lockFactory=org.apache.lucene.store.NativeFSLockFactory@56889e1c),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1037236 T1818 C678 P52431 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1037237 T1818 C678 P52431 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@32eeca40 lockFactory=org.apache.lucene.store.NativeFSLockFactory@56889e1c),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@32eeca40 lockFactory=org.apache.lucene.store.NativeFSLockFactory@56889e1c),segFN=segments_2,generation=2}
[junit4:junit4] 2> 1037238 T1818 C678 P52431 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 1037238 T1818 C678 P52431 oass.SolrIndexSearcher.<init> Opening Searcher@2a1fa744 realtime
[junit4:junit4] 2> 1037239 T1818 C678 P52431 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 1037239 T1818 C678 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 52
[junit4:junit4] 2> 1037241 T1860 C677 P52437 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 1037241 T1860 C677 P52437 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 1037251 T1821 C678 P52431 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 1037252 T1821 C678 P52431 oasc.SolrCore.execute [collection1] webapp=/df path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=6
[junit4:junit4] 2> 1037253 T1860 C677 P52437 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 1037253 T1860 C677 P52437 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 1037254 T1860 C677 P52437 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 1037257 T1816 C678 P52431 oasc.SolrCore.execute [collection1] webapp=/df path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 1037258 T1860 C677 P52437 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 1037270 T1860 C677 P52437 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty3/index.20130528000516421
[junit4:junit4] 2> 1037270 T1860 C677 P52437 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@1511c6ee lockFactory=org.apache.lucene.store.NativeFSLockFactory@51c718ce) fullCopy=false
[junit4:junit4] 2> 1037273 T1820 C678 P52431 oasc.SolrCore.execute [collection1] webapp=/df path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 1037286 T1860 C677 P52437 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 1037309 T1860 C677 P52437 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 1037310 T1860 C677 P52437 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 1037315 T1860 C677 P52437 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@14414fe5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@190ba1c3),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@14414fe5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@190ba1c3),segFN=segments_2,generation=2}
[junit4:junit4] 2> 1037354 T1860 C677 P52437 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 1037358 T1860 C677 P52437 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 1037359 T1860 C677 P52437 oass.SolrIndexSearcher.<init> Opening Searcher@44fbac51 main
[junit4:junit4] 2> 1037361 T1859 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@44fbac51 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 1037362 T1860 C677 P52437 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty3/index.20130528000516421 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty3/index.20130528000516421;done=true>>]
[junit4:junit4] 2> 1037363 T1860 C677 P52437 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty3/index.20130528000516421
[junit4:junit4] 2> 1037363 T1860 C677 P52437 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty3/index.20130528000516421
[junit4:junit4] 2> 1037363 T1860 C677 P52437 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 1037364 T1860 C677 P52437 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 1037364 T1860 C677 P52437 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 1037364 T1860 C677 P52437 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1037367 T1860 C677 P52437 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 1037840 T1875 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 1037845 T1875 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 1037848 T1875 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1037857 T1875 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1037952 T1806 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1037954 T1806 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:52437_df_collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52437_df",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52437/df"}
[junit4:junit4] 2> 1037965 T1875 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1037965 T1875 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369713913270/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty4/
[junit4:junit4] 2> 1037966 T1875 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ed56fe5
[junit4:junit4] 2> 1037968 T1875 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1037977 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1037977 T1857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1037978 T1825 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1037978 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1037978 T1841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1037981 T1875 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty4
[junit4:junit4] 2> 1037981 T1875 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty4/index/
[junit4:junit4] 2> 1037978 T1874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1037982 T1875 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 1037990 T1875 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty4/index
[junit4:junit4] 2> 1037998 T1875 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4e28ed8d lockFactory=org.apache.lucene.store.NativeFSLockFactory@2b25f2be),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1037999 T1875 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1038004 T1875 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 1038004 T1875 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 1038005 T1875 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 1038006 T1875 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 1038007 T1875 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 1038008 T1875 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 1038009 T1875 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 1038010 T1875 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 1038011 T1875 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 1038031 T1875 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 1038040 T1875 oass.SolrIndexSearcher.<init> Opening Searcher@79140b4d main
[junit4:junit4] 2> 1038041 T1875 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty4/tlog
[junit4:junit4] 2> 1038042 T1875 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 1038043 T1875 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 1038048 T1878 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79140b4d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 1038053 T1875 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1038054 T1875 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1039508 T1806 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1039510 T1806 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52441_df",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52441/df"}
[junit4:junit4] 2> 1039511 T1806 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 1039511 T1806 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 1039529 T1874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1039530 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1039533 T1841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1039534 T1857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1039535 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1039536 T1825 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1040061 T1875 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 1040062 T1875 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52441/df collection:collection1 shard:shard2
[junit4:junit4] 2> 1040075 T1875 oasc.ZkController.register We are http://127.0.0.1:52441/df/collection1/ and leader is http://127.0.0.1:52434/df/collection1/
[junit4:junit4] 2> 1040075 T1875 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52441/df
[junit4:junit4] 2> 1040076 T1875 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 1040080 T1875 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C679 name=collection1 org.apache.solr.core.SolrCore@6c46adce url=http://127.0.0.1:52441/df/collection1 node=127.0.0.1:52441_df C679_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:52441_df, base_url=http://127.0.0.1:52441/df}
[junit4:junit4] 2> 1040086 T1879 C679 P52441 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 1040087 T1879 C679 P52441 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 1040088 T1875 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1040088 T1879 C679 P52441 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 1040089 T1879 C679 P52441 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1040093 T1783 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4:junit4] 2> 1040094 T1783 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 1040093 T1879 C679 P52441 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1040094 T1783 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1040140 T1832 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:52441_df_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 1040143 T1783 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1040145 T1783 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 1040146 T1783 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 1040148 T1783 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1041079 T1806 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1041083 T1806 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:52441_df_collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52441_df",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52441/df"}
[junit4:junit4] 2> 1041101 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1041101 T1857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1041102 T1825 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1041102 T1874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1041102 T1841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1041102 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1041142 T1832 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:52441_df_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 1041142 T1832 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:52441_df_collection1&state=recovering&nodeName=127.0.0.1:52441_df&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4] 2> 1041158 T1783 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1042164 T1783 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C679_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:52441_df, base_url=http://127.0.0.1:52441/df}
[junit4:junit4] 2> 1043145 T1879 C679 P52441 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:52434/df/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 1043145 T1879 C679 P52441 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52441/df START replicas=[http://127.0.0.1:52434/df/collection1/] nUpdates=100
[junit4:junit4] 2> 1043146 T1879 C679 P52441 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 1043146 T1879 C679 P52441 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 1043146 T1879 C679 P52441 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 1043147 T1879 C679 P52441 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 1043147 T1879 C679 P52441 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 1043147 T1879 C679 P52441 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:52434/df/collection1/. core=collection1
[junit4:junit4] 2> 1043148 T1879 C679 P52441 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 1043198 T1783 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C680 name=collection1 org.apache.solr.core.SolrCore@1e327923 url=http://127.0.0.1:52434/df/collection1 node=127.0.0.1:52434_df C680_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52434_df, base_url=http://127.0.0.1:52434/df, leader=true}
[junit4:junit4] 2> 1043213 T1835 C680 P52434 oasc.SolrCore.execute [collection1] webapp=/df path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=2
[junit4:junit4] 2> 1043219 T1836 C680 P52434 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 1043226 T1836 C680 P52434 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f255dc5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@d7b5bd4),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1043232 T1836 C680 P52434 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1043239 T1836 C680 P52434 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f255dc5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@d7b5bd4),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f255dc5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@d7b5bd4),segFN=segments_2,generation=2}
[junit4:junit4] 2> 1043241 T1836 C680 P52434 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 1043247 T1836 C680 P52434 oass.SolrIndexSearcher.<init> Opening Searcher@397f0b97 realtime
[junit4:junit4] 2> 1043247 T1836 C680 P52434 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 1043248 T1836 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 29
[junit4:junit4] 2> 1043250 T1879 C679 P52441 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 1043251 T1879 C679 P52441 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 1043268 T1833 C680 P52434 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 1043269 T1833 C680 P52434 oasc.SolrCore.execute [collection1] webapp=/df path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=3
[junit4:junit4] 2> 1043271 T1879 C679 P52441 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 1043271 T1879 C679 P52441 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 1043272 T1879 C679 P52441 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 1043279 T1834 C680 P52434 oasc.SolrCore.execute [collection1] webapp=/df path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 1043281 T1879 C679 P52441 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 1043286 T1879 C679 P52441 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty4/index.20130528000522445
[junit4:junit4] 2> 1043287 T1879 C679 P52441 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@40c459b9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e78d4a) fullCopy=false
[junit4:junit4] 2> 1043295 T1832 C680 P52434 oasc.SolrCore.execute [collection1] webapp=/df path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 1043309 T1879 C679 P52441 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 1043332 T1879 C679 P52441 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 1043332 T1879 C679 P52441 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 1043336 T1879 C679 P52441 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4e28ed8d lockFactory=org.apache.lucene.store.NativeFSLockFactory@2b25f2be),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4e28ed8d lockFactory=org.apache.lucene.store.NativeFSLockFactory@2b25f2be),segFN=segments_2,generation=2}
[junit4:junit4] 2> 1043337 T1879 C679 P52441 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 1043347 T1879 C679 P52441 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 1043358 T1879 C679 P52441 oass.SolrIndexSearcher.<init> Opening Searcher@7a6d91c5 main
[junit4:junit4] 2> 1043361 T1878 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7a6d91c5 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 1043362 T1879 C679 P52441 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty4/index.20130528000522445 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty4/index.20130528000522445;done=true>>]
[junit4:junit4] 2> 1043362 T1879 C679 P52441 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty4/index.20130528000522445
[junit4:junit4] 2> 1043364 T1879 C679 P52441 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty4/index.20130528000522445
[junit4:junit4] 2> 1043368 T1879 C679 P52441 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 1043368 T1879 C679 P52441 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 1043369 T1879 C679 P52441 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 1043369 T1879 C679 P52441 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1043375 T1879 C679 P52441 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 1044148 T1806 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1044153 T1806 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:52441_df_collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52441_df",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52441/df"}
[junit4:junit4] 2> 1044198 T1874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1044198 T1805 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1044199 T1857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1044200 T1841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1044201 T1825 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1044199 T1812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 1044225 T1783 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1044229 T1783 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C681 name=collection1 org.apache.solr.core.SolrCore@4adf0ee9 url=http://127.0.0.1:52427/df/collection1 node=127.0.0.1:52427_df C681_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:52427_df, base_url=http://127.0.0.1:52427/df, leader=true}
[junit4:junit4] 2> 1044263 T1796 C681 P52427 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 1044276 T1796 C681 P52427 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@56d0ddb8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c24cb1b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 1044277 T1796 C681 P52427 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 1044292 T1796 C681 P52427 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@56d0ddb8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c24cb1b),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@56d0ddb8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2c24cb1b),segFN=segments_2,generation=2}
[junit4:junit4] 2> 1044293 T1796 C681 P52427 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 1044296 T1796 C681 P52427 oass.SolrIndexSearcher.<init> Opening Searcher@67c5c23a main
[junit4:junit4] 2> 1044307 T1809 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@67c5c23a main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 1044309 T1796 C681 P52427 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 1044317 T1796 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 53
[junit4:junit4] 2> ASYNC NEW_CORE C682 name=collection1 org.apache.solr.core.SolrCore@3383994a url=http://127.0.0.1:52431/df/collection1 node=127.0.0.1:52431_df C682_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52431_df, base_url=http://127.0.0.1:52431/df, leader=true}
[junit4:junit4] 2> 1044370 T1820 C682 P52431 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 1044398 T1820 C682 P52431 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@32eeca40 lockFactory=org.apache.lucene.store.NativeFSLockFactory@56889e1c),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@32eeca40 lockFactory=org.apache.lucene.store.NativeFSLockFactory@56889e1c),segFN=segments_3,generation=3}
[junit4:junit4] 2> 1044400 T1820 C682 P52431 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 1044401 T1820 C682 P52431 oass.SolrIndexSearcher.<init> Opening Searcher@111b8549 main
[junit4:junit4] 2> 1044402 T1820 C682 P52431 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 1044405 T1827 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@111b8549 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 1044406 T1820 C682 P52431 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:52437/df/collection1/, StdNode: http://127.0.0.1:52434/df/collection1/, StdNode: http://127.0.0.1:52441/df/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 1044436 T1837 C680 P52434 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 1044477 T1837 C680 P52434 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f255dc5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@d7b5bd4),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f255dc5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@d7b5bd4),segFN=segments_3,generation=3}
[junit4:junit4] 2> 1044482 T1837 C680 P52434 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 1044484 T1837 C680 P52434 oass.SolrIndexSearcher.<init> Opening Searcher@7d622e9d main
[junit4:junit4] 2> 1044486 T1837 C680 P52434 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 1044487 T1843 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7d622e9d main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> C679_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52441_df, base_url=http://127.0.0.1:52441/df}
[junit4:junit4] 2> 1044491 T1869 C679 P52441 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 1044495 T1837 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 56
[junit4:junit4] 2> ASYNC NEW_CORE C683 name=collection1 org.apache.solr.core.SolrCore@7b3bd807 url=http://127.0.0.1:52437/df/collection1 node=127.0.0.1:52437_df C683_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52437_df, base_url=http://127.0.0.1:52437/df}
[junit4:junit4] 2> 1044502 T1850 C683 P52437 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 1044513 T1869 C679 P52441 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4e28ed8d lockFactory=org.apache.lucene.store.NativeFSLockFactory@2b25f2be),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4e28ed8d lockFactory=org.apache.lucene.store.NativeFSLockFactory@2b25f2be),segFN=segments_3,generation=3}
[junit4:junit4] 2> 1044514 T1869 C679 P52441 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 1044516 T1869 C679 P52441 oass.SolrIndexSearcher.<init> Opening Searcher@209d1789 main
[junit4:junit4] 2> 1044517 T1869 C679 P52441 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 1044518 T1878 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@209d1789 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 1044521 T1869 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 30
[junit4:junit4] 2> 1044537 T1850 C683 P52437 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@14414fe5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@190ba1c3),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@14414fe5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@190ba1c3),segFN=segments_3,generation=3}
[junit4:junit4] 2> 1044544 T1850 C683 P52437 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 1044552 T1850 C683 P52437 oass.SolrIndexSearcher.<init> Opening Searcher@54cfa13a main
[junit4:junit4] 2> 1044558 T1859 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@54cfa13a main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 1044558 T1850 C683 P52437 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 1044559 T1850 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 57
[junit4:junit4] 2> 1044560 T1820 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 190
[junit4:junit4] 2> 1044565 T1783 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1044578 T1818 C682 P52431 oasc.SolrCore.execute [collection1] webapp=/df path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 1044587 T1851 C683 P52437 oasc.SolrCore.execute [collection1] webapp=/df path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 1044599 T1834 C680 P52434 oasc.SolrCore.execute [collection1] webapp=/df path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 1044607 T1870 C679 P52441 oasc.SolrCore.execute [collection1] webapp=/df path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 1046618 T1796 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1436249149435543552)} 0 4
[junit4:junit4] 2> 1046661 T1870 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={update.distrib=FROMLEADER&_version_=-1436249149450223616&update.from=http://127.0.0.1:52434/df/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1436249149450223616)} 0 17
[junit4:junit4] 2> 1046671 T1851 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={update.distrib=FROMLEADER&_version_=-1436249149468049408&update.from=http://127.0.0.1:52431/df/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1436249149468049408)} 0 4
[junit4:junit4] 2> 1046673 T1818 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1436249149468049408)} 0 27
[junit4:junit4] 2> 1046677 T1833 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1436249149450223616)} 0 49
[junit4:junit4] 2> 1046689 T1797 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[0 (1436249149507895296)]} 0 5
[junit4:junit4] 2> 1046734 T1852 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1436249149532012544)]} 0 11
[junit4:junit4] 2> 1046738 T1819 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1436249149532012544)]} 0 32
[junit4:junit4] 2> 1046741 T1834 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[0]} 0 46
[junit4:junit4] 2> 1046752 T1798 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[1 (1436249149577101312)]} 0 3
[junit4:junit4] 2> 1046778 T1853 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1436249149587587072)]} 0 10
[junit4:junit4] 2> 1046781 T1821 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[1 (1436249149587587072)]} 0 21
[junit4:junit4] 2> 1046794 T1800 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[2 (1436249149623238656)]} 0 1
[junit4:junit4] 2> 1046830 T1865 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1436249149647355904)]} 0 4
[junit4:junit4] 2> 1046832 T1832 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[2 (1436249149647355904)]} 0 15
[junit4:junit4] 2> 1046836 T1816 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[2]} 0 25
[junit4:junit4] 2> 1046844 T1801 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[3 (1436249149674618880)]} 0 1
[junit4:junit4] 2> 1046883 T1866 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[3 (1436249149701881856)]} 0 2
[junit4:junit4] 2> 1046886 T1835 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[3 (1436249149701881856)]} 0 19
[junit4:junit4] 2> 1046889 T1820 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[3]} 0 35
[junit4:junit4] 2> 1046895 T1796 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[4 (1436249149728096256)]} 0 2
[junit4:junit4] 2> 1046922 T1848 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[4 (1436249149748019200)]} 0 1
[junit4:junit4] 2> 1046924 T1817 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[4 (1436249149748019200)]} 0 13
[junit4:junit4] 2> 1046927 T1836 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[4]} 0 18
[junit4:junit4] 2> 1046932 T1797 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[5 (1436249149767942144)]} 0 0
[junit4:junit4] 2> 1046946 T1867 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[5 (1436249149777379328)]} 0 1
[junit4:junit4] 2> 1046948 T1837 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[5 (1436249149777379328)]} 0 8
[junit4:junit4] 2> 1046954 T1799 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[6 (1436249149789962240)]} 0 1
[junit4:junit4] 2> 1046981 T1868 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[6 (1436249149811982336)]} 0 4
[junit4:junit4] 2> 1046983 T1833 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[6 (1436249149811982336)]} 0 10
[junit4:junit4] 2> 1046985 T1818 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[6]} 0 25
[junit4:junit4] 2> 1046989 T1798 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[7 (1436249149827710976)]} 0 0
[junit4:junit4] 2> 1047004 T1869 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[7 (1436249149836099584)]} 0 1
[junit4:junit4] 2> 1047006 T1834 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[7 (1436249149836099584)]} 0 9
[junit4:junit4] 2> 1047009 T1819 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[7]} 0 15
[junit4:junit4] 2> 1047014 T1800 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[8 (1436249149852876800)]} 0 1
[junit4:junit4] 2> 1047027 T1849 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[8 (1436249149861265408)]} 0 1
[junit4:junit4] 2> 1047028 T1816 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[8 (1436249149861265408)]} 0 7
[junit4:junit4] 2> 1047030 T1835 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[8]} 0 12
[junit4:junit4] 2> 1047040 T1801 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[9 (1436249149880139776)]} 0 1
[junit4:junit4] 2> 1047053 T1870 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[9 (1436249149885382656)]} 0 4
[junit4:junit4] 2> 1047058 T1836 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[9 (1436249149885382656)]} 0 14
[junit4:junit4] 2> 1047070 T1796 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[10 (1436249149912645632)]} 0 1
[junit4:junit4] 2> 1047080 T1851 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10 (1436249149917888512)]} 0 1
[junit4:junit4] 2> 1047083 T1820 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[10 (1436249149917888512)]} 0 8
[junit4:junit4] 2> 1047090 T1797 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[11 (1436249149931520000)]} 0 2
[junit4:junit4] 2> 1047104 T1852 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[11 (1436249149943054336)]} 0 1
[junit4:junit4] 2> 1047106 T1817 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[11 (1436249149943054336)]} 0 7
[junit4:junit4] 2> 1047108 T1837 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[11]} 0 13
[junit4:junit4] 2> 1047116 T1799 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[12 (1436249149959831552)]} 0 1
[junit4:junit4] 2> 1047126 T1853 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1436249149966123008)]} 0 1
[junit4:junit4] 2> 1047129 T1818 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[12 (1436249149966123008)]} 0 8
[junit4:junit4] 2> 1047135 T1798 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[13 (1436249149979754496)]} 0 2
[junit4:junit4] 2> 1047153 T1848 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1436249149987094528)]} 0 6
[junit4:junit4] 2> 1047156 T1821 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[13 (1436249149987094528)]} 0 16
[junit4:junit4] 2> 1047164 T1800 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[14 (1436249150011211776)]} 0 0
[junit4:junit4] 2> 1047184 T1850 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1436249150025891840)]} 0 1
[junit4:junit4] 2> 1047193 T1819 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[14 (1436249150025891840)]} 0 15
[junit4:junit4] 2> 1047204 T1833 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[14]} 0 34
[junit4:junit4] 2> 1047211 T1796 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[15 (1436249150060494848)]} 0 1
[junit4:junit4] 2> 1047231 T1849 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1436249150073077760)]} 0 2
[junit4:junit4] 2> 1047240 T1816 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[15 (1436249150073077760)]} 0 17
[junit4:junit4] 2> 1047244 T1834 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[15]} 0 27
[junit4:junit4] 2> 1047260 T1797 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[16 (1436249150108729344)]} 0 4
[junit4:junit4] 2> 1047287 T1851 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1436249150129700864)]} 0 5
[junit4:junit4] 2> 1047299 T1820 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[16 (1436249150129700864)]} 0 24
[junit4:junit4] 2> 1047302 T1832 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[16]} 0 31
[junit4:junit4] 2> 1047309 T1799 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[17 (1436249150162206720)]} 0 1
[junit4:junit4] 2> 1047326 T1866 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1436249150170595328)]} 0 1
[junit4:junit4] 2> 1047336 T1835 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[17 (1436249150170595328)]} 0 20
[junit4:junit4] 2> 1047344 T1798 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[18 (1436249150198906880)]} 0 0
[junit4:junit4] 2> 1047354 T1867 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1436249150206246912)]} 0 1
[junit4:junit4] 2> 1047356 T1836 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[18 (1436249150206246912)]} 0 7
[junit4:junit4] 2> 1047361 T1801 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[19 (1436249150217781248)]} 0 1
[junit4:junit4] 2> 1047369 T1868 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1436249150223024128)]} 0 0
[junit4:junit4] 2> 1047371 T1837 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[19 (1436249150223024128)]} 0 6
[junit4:junit4] 2> 1047375 T1800 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[20 (1436249150232461312)]} 0 0
[junit4:junit4] 2> 1047388 T1852 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1436249150241898496)]} 0 1
[junit4:junit4] 2> 1047394 T1817 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[20 (1436249150241898496)]} 0 11
[junit4:junit4] 2> 1047401 T1833 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[20]} 0 21
[junit4:junit4] 2> 1047408 T1796 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[21 (1436249150266015744)]} 0 1
[junit4:junit4] 2> 1047423 T1869 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1436249150276501504)]} 0 2
[junit4:junit4] 2> 1047424 T1834 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[21 (1436249150276501504)]} 0 7
[junit4:junit4] 2> 1047426 T1821 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[21]} 0 13
[junit4:junit4] 2> 1047432 T1797 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[22 (1436249150291181568)]} 0 1
[junit4:junit4] 2> 1047448 T1853 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1436249150301667328)]} 0 3
[junit4:junit4] 2> 1047449 T1819 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[22 (1436249150301667328)]} 0 8
[junit4:junit4] 2> 1047452 T1835 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[22]} 0 15
[junit4:junit4] 2> 1047459 T1799 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[23 (1436249150320541696)]} 0 1
[junit4:junit4] 2> 1047473 T1865 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1436249150329978880)]} 0 1
[junit4:junit4] 2> 1047475 T1836 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[23 (1436249150329978880)]} 0 7
[junit4:junit4] 2> 1047477 T1816 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[23]} 0 13
[junit4:junit4] 2> 1047483 T1798 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[24 (1436249150344658944)]} 0 2
[junit4:junit4] 2> 1047510 T1850 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1436249150368776192)]} 0 1
[junit4:junit4] 2> 1047513 T1820 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[24 (1436249150368776192)]} 0 9
[junit4:junit4] 2> 1047515 T1837 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[24]} 0 22
[junit4:junit4] 2> 1047521 T1801 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[25 (1436249150384504832)]} 0 1
[junit4:junit4] 2> 1047535 T1866 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1436249150394990592)]} 0 1
[junit4:junit4] 2> 1047537 T1833 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[25 (1436249150394990592)]} 0 7
[junit4:junit4] 2> 1047540 T1817 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[25]} 0 14
[junit4:junit4] 2> 1047546 T1800 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[26 (1436249150410719232)]} 0 2
[junit4:junit4] 2> 1047559 T1867 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1436249150421204992)]} 0 1
[junit4:junit4] 2> 1047562 T1832 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[26 (1436249150421204992)]} 0 8
[junit4:junit4] 2> 1047564 T1818 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[26]} 0 14
[junit4:junit4] 2> 1047569 T1796 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[27 (1436249150435885056)]} 0 1
[junit4:junit4] 2> 1047597 T1849 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1436249150454759424)]} 0 1
[junit4:junit4] 2> 1047598 T1821 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[27 (1436249150454759424)]} 0 11
[junit4:junit4] 2> 1047600 T1834 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[27]} 0 25
[junit4:junit4] 2> 1047609 T1799 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[28 (1436249150476779520)]} 0 1
[junit4:junit4] 2> 1047619 T1851 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1436249150483070976)]} 0 1
[junit4:junit4] 2> 1047621 T1819 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[28 (1436249150483070976)]} 0 7
[junit4:junit4] 2> 1047633 T1798 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[29 (1436249150501945344)]} 0 1
[junit4:junit4] 2> 1047646 T1868 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1436249150512431104)]} 0 1
[junit4:junit4] 2> 1047648 T1835 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[29 (1436249150512431104)]} 0 7
[junit4:junit4] 2> 1047650 T1816 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[29]} 0 12
[junit4:junit4] 2> 1047656 T1801 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[30 (1436249150526062592)]} 0 1
[junit4:junit4] 2> 1047666 T1869 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1436249150532354048)]} 0 2
[junit4:junit4] 2> 1047667 T1836 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[30 (1436249150532354048)]} 0 7
[junit4:junit4] 2> 1047672 T1800 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[31 (1436249150543888384)]} 0 1
[junit4:junit4] 2> 1047697 T1870 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1436249150556471296)]} 0 1
[junit4:junit4] 2> 1047699 T1837 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[31 (1436249150556471296)]} 0 18
[junit4:junit4] 2> 1047705 T1797 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[32 (1436249150577442816)]} 0 1
[junit4:junit4] 2> 1047715 T1852 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1436249150583734272)]} 0 1
[junit4:junit4] 2> 1047717 T1820 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[32 (1436249150583734272)]} 0 7
[junit4:junit4] 2> 1047728 T1796 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[33 (1436249150601560064)]} 0 1
[junit4:junit4] 2> 1047752 T1865 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1436249150615191552)]} 0 8
[junit4:junit4] 2> 1047755 T1833 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[33 (1436249150615191552)]} 0 16
[junit4:junit4] 2> 1047757 T1817 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[33]} 0 21
[junit4:junit4] 2> 1047763 T1799 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[34 (1436249150638260224)]} 0 1
[junit4:junit4] 2> 1047776 T1848 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1436249150647697408)]} 0 1
[junit4:junit4] 2> 1047778 T1818 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[34 (1436249150647697408)]} 0 7
[junit4:junit4] 2> 1047780 T1832 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[34]} 0 13
[junit4:junit4] 2> 1047786 T1798 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[35 (1436249150662377472)]} 0 1
[junit4:junit4] 2> 1047807 T1853 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1436249150674960384)]} 0 1
[junit4:junit4] 2> 1047808 T1821 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[35 (1436249150674960384)]} 0 14
[junit4:junit4] 2> 1047810 T1834 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[35]} 0 20
[junit4:junit4] 2> 1047816 T1801 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[36 (1436249150694883328)]} 0 1
[junit4:junit4] 2> 1047829 T1850 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1436249150703271936)]} 0 1
[junit4:junit4] 2> 1047832 T1816 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[36 (1436249150703271936)]} 0 8
[junit4:junit4] 2> 1047838 T1800 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[37 (1436249150716903424)]} 0 2
[junit4:junit4] 2> 1047850 T1866 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1436249150725292032)]} 0 1
[junit4:junit4] 2> 1047852 T1835 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[37 (1436249150725292032)]} 0 8
[junit4:junit4] 2> 1047858 T1797 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[38 (1436249150737874944)]} 0 1
[junit4:junit4] 2> 1047874 T1849 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[38 (1436249150749409280)]} 0 0
[junit4:junit4] 2> 1047876 T1820 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[38 (1436249150749409280)]} 0 10
[junit4:junit4] 2> 1047889 T1796 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[39 (1436249150771429376)]} 0 1
[junit4:junit4] 2> 1047902 T1867 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[39 (1436249150779817984)]} 0 1
[junit4:junit4] 2> 1047906 T1837 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[39 (1436249150779817984)]} 0 9
[junit4:junit4] 2> 1047909 T1817 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[39]} 0 16
[junit4:junit4] 2> 1047916 T1799 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[40 (1436249150798692352)]} 0 1
[junit4:junit4] 2> 1047928 T1852 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[40 (1436249150808129536)]} 0 1
[junit4:junit4] 2> 1047934 T1818 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[40 (1436249150808129536)]} 0 11
[junit4:junit4] 2> 1047940 T1833 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[40]} 0 20
[junit4:junit4] 2> 1047946 T1798 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[41 (1436249150831198208)]} 0 1
[junit4:junit4] 2> 1047963 T1848 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[41 (1436249150842732544)]} 0 1
[junit4:junit4] 2> 1047966 T1821 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[41 (1436249150842732544)]} 0 9
[junit4:junit4] 2> 1047967 T1832 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[41]} 0 13
[junit4:junit4] 2> 1047972 T1801 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[42 (1436249150858461184)]} 0 0
[junit4:junit4] 2> 1047991 T1853 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[42 (1436249150863704064)]} 0 2
[junit4:junit4] 2> 1047993 T1819 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[42 (1436249150863704064)]} 0 16
[junit4:junit4] 2> 1048001 T1797 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[43 (1436249150888869888)]} 0 1
[junit4:junit4] 2> 1048021 T1850 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[43 (1436249150903549952)]} 0 1
[junit4:junit4] 2> 1048023 T1816 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[43 (1436249150903549952)]} 0 12
[junit4:junit4] 2> 1048029 T1796 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[44 (1436249150917181440)]} 0 1
[junit4:junit4] 2> 1048049 T1851 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[44 (1436249150933958656)]} 0 1
[junit4:junit4] 2> 1048051 T1820 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[44 (1436249150933958656)]} 0 7
[junit4:junit4] 2> 1048053 T1834 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[44]} 0 17
[junit4:junit4] 2> 1048059 T1799 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[45 (1436249150948638720)]} 0 1
[junit4:junit4] 2> 1048071 T1869 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[45 (1436249150957027328)]} 0 1
[junit4:junit4] 2> 1048073 T1836 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[45 (1436249150957027328)]} 0 8
[junit4:junit4] 2> 1048078 T1798 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[46 (1436249150968561664)]} 0 1
[junit4:junit4] 2> 1048091 T1849 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[46 (1436249150977998848)]} 0 1
[junit4:junit4] 2> 1048093 T1817 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[46 (1436249150977998848)]} 0 7
[junit4:junit4] 2> 1048095 T1835 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[46]} 0 13
[junit4:junit4] 2> 1048101 T1801 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[47 (1436249150992678912)]} 0 1
[junit4:junit4] 2> 1048111 T1852 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[47 (1436249151000018944)]} 0 1
[junit4:junit4] 2> 1048113 T1818 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[47 (1436249151000018944)]} 0 7
[junit4:junit4] 2> 1048119 T1800 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[48 (1436249151011553280)]} 0 2
[junit4:junit4] 2> 1048134 T1848 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[48 (1436249151022039040)]} 0 1
[junit4:junit4] 2> 1048136 T1821 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[48 (1436249151022039040)]} 0 8
[junit4:junit4] 2> 1048145 T1797 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[49 (1436249151038816256)]} 0 1
[junit4:junit4] 2> 1048154 T1853 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[49 (1436249151044059136)]} 0 1
[junit4:junit4] 2> 1048155 T1819 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[49 (1436249151044059136)]} 0 6
[junit4:junit4] 2> 1048160 T1796 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[50 (1436249151055593472)]} 0 0
[junit4:junit4] 2> 1048169 T1850 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[50 (1436249151060836352)]} 0 0
[junit4:junit4] 2> 1048171 T1816 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[50 (1436249151060836352)]} 0 7
[junit4:junit4] 2> 1048177 T1799 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[51 (1436249151072370688)]} 0 1
[junit4:junit4] 2> 1048201 T1851 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[51 (1436249151094390784)]} 0 1
[junit4:junit4] 2> 1048204 T1820 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[51 (1436249151094390784)]} 0 8
[junit4:junit4] 2> 1048206 T1837 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[51]} 0 19
[junit4:junit4] 2> 1048213 T1798 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[52 (1436249151109070848)]} 0 1
[junit4:junit4] 2> 1048240 T1849 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[52 (1436249151131090944)]} 0 1
[junit4:junit4] 2> 1048242 T1817 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[52 (1436249151131090944)]} 0 10
[junit4:junit4] 2> 1048246 T1801 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[53 (1436249151145771008)]} 0 1
[junit4:junit4] 2> 1048257 T1870 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[53 (1436249151153111040)]} 0 1
[junit4:junit4] 2> 1048258 T1833 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[53 (1436249151153111040)]} 0 5
[junit4:junit4] 2> 1048260 T1821 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[53]} 0 10
[junit4:junit4] 2> 1048268 T1800 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[54 (1436249151168839680)]} 0 0
[junit4:junit4] 2> 1048279 T1852 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[54 (1436249151175131136)]} 0 1
[junit4:junit4] 2> 1048281 T1819 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[54 (1436249151175131136)]} 0 8
[junit4:junit4] 2> 1048287 T1797 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[55 (1436249151187714048)]} 0 1
[junit4:junit4] 2> 1048299 T1853 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[55 (1436249151197151232)]} 0 0
[junit4:junit4] 2> 1048301 T1816 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[55 (1436249151197151232)]} 0 7
[junit4:junit4] 2> 1048303 T1834 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[55]} 0 12
[junit4:junit4] 2> 1048309 T1796 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[56 (1436249151211831296)]} 0 1
[junit4:junit4] 2> 1048328 T1865 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[56 (1436249151217074176)]} 0 4
[junit4:junit4] 2> 1048330 T1836 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[56 (1436249151217074176)]} 0 17
[junit4:junit4] 2> 1048341 T1799 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[57 (1436249151244337152)]} 0 1
[junit4:junit4] 2> 1048357 T1867 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[57 (1436249151257968640)]} 0 1
[junit4:junit4] 2> 1048359 T1835 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[57 (1436249151257968640)]} 0 7
[junit4:junit4] 2> 1048361 T1820 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[57]} 0 12
[junit4:junit4] 2> 1048366 T1798 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[58 (1436249151271600128)]} 0 1
[junit4:junit4] 2> 1048380 T1868 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[58 (1436249151281037312)]} 0 1
[junit4:junit4] 2> 1048382 T1837 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[58 (1436249151281037312)]} 0 7
[junit4:junit4] 2> 1048385 T1818 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[58]} 0 14
[junit4:junit4] 2> 1048390 T1801 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[59 (1436249151296765952)]} 0 1
[junit4:junit4] 2> 1048404 T1850 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[59 (1436249151307251712)]} 0 1
[junit4:junit4] 2> 1048407 T1817 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[59 (1436249151307251712)]} 0 8
[junit4:junit4] 2> 1048410 T1832 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[59]} 0 15
[junit4:junit4] 2> 1048417 T1797 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[60 (1436249151322980352)]} 0 2
[junit4:junit4] 2> 1048436 T1869 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[60 (1436249151335563264)]} 0 1
[junit4:junit4] 2> 1048437 T1833 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[60 (1436249151335563264)]} 0 11
[junit4:junit4] 2> 1048446 T1796 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[61 (1436249151354437632)]} 0 0
[junit4:junit4] 2> 1048458 T1851 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[61 (1436249151362826240)]} 0 0
[junit4:junit4] 2> 1048460 T1821 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[61 (1436249151362826240)]} 0 7
[junit4:junit4] 2> 1048462 T1834 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[61]} 0 12
[junit4:junit4] 2> 1048469 T1799 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[62 (1436249151378554880)]} 0 1
[junit4:junit4] 2> 1048478 T1870 C679 P52441 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[62 (1436249151383797760)]} 0 1
[junit4:junit4] 2> 1048480 T1836 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[62 (1436249151383797760)]} 0 7
[junit4:junit4] 2> 1048486 T1798 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[63 (1436249151396380672)]} 0 1
[junit4:junit4] 2> 1048501 T1849 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[63 (1436249151406866432)]} 0 1
[junit4:junit4] 2> 1048503 T1819 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52434/df/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[63 (1436249151406866432)]} 0 8
[junit4:junit4] 2> 1048506 T1835 C680 P52434 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[63]} 0 15
[junit4:junit4] 2> 1048521 T1800 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[64 (1436249151433080832)]} 0 1
[junit4:junit4] 2> 1048541 T1848 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[64 (1436249151439372288)]} 0 1
[junit4:junit4] 2> 1048543 T1816 C682 P52431 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[64 (1436249151439372288)]} 0 17
[junit4:junit4] 2> 1048549 T1801 C681 P52427 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={wt=javabin&version=2} {add=[65 (1436249151462440960)]} 0 1
[junit4:junit4] 2> 1048559 T1852 C683 P52437 oasup.LogUpdateProcessor.finish [collection1] webapp=/df path=/update params={distrib.from=http://127.0.0.1:52431/df/collection1/&update.dist
[...truncated too long message...]
junit4] 2> "shard":"shard1_0",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":null,
[junit4:junit4] 2> "core":"collection1_shard1_0_replica1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52437_df",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52437/df"}
[junit4:junit4] 2> 1089541 T1943 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:52441_df_collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:52441_df",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52441/df"}
[junit4:junit4] 2> 1089556 T1874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1090499 T1783 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1090500 T1783 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1090505 T1783 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@6c46adce
[junit4:junit4] 2> 1090508 T1783 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=4,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=272,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 1090511 T1783 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 1090512 T1783 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 1090513 T1783 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 1090519 T1783 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 1090520 T1783 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 1090521 T1783 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty4/index;done=false>>]
[junit4:junit4] 2> 1090522 T1783 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty4/index
[junit4:junit4] 2> 1090523 T1783 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty4;done=false>>]
[junit4:junit4] 2> 1090524 T1783 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001/jetty4
[junit4:junit4] 2> 1090529 T1943 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89765569234206732-127.0.0.1:52441_df-n_0000000004) am no longer a leader.
[junit4:junit4] 2> 1090556 T1783 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/df,null}
[junit4:junit4] 2> 1090671 T1783 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 1090679 T1857 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1090680 T1857 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 1090681 T1857 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1090682 T1783 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52424 52424
[junit4:junit4] 2> 1090792 T1783 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 52427
[junit4:junit4] 2> !!!! WARNING: best effort to remove /Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369713886001 FAILED !!!!!
[junit4:junit4] 2> 1090794 T1783 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 1090795 T1783 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52424 52424
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeyShardSplitTest -Dtests.method=testDistribSearch -Dtests.seed=C345B394745EAB26 -Dtests.slow=true -Dtests.locale=be_BY -Dtests.timezone=America/Guadeloupe -Dtests.file.encoding=US-ASCII
[junit4:junit4] ERROR 84.0s | ChaosMonkeyShardSplitTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:52431/df returned non ok status:500, message:Server Error
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([C345B394745EAB26:42A33D8C0301CB1A]:0)
[junit4:junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
[junit4:junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4] > at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
[junit4:junit4] > at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 1090823 T1783 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 84011 T1782 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 1091837 T1874 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 1091838 T1874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 1091838 T1874 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=be_BY, timezone=America/Guadeloupe
[junit4:junit4] 2> NOTE: Mac OS X 10.8.3 x86_64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=2,threads=4,free=154064328,total=325910528
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestPerFieldSimilarity, TestBinaryField, TestCSVResponseWriter, IndexReaderFactoryTest, SpatialFilterTest, BasicZkTest, TestCollationField, ShardSplitTest, CurrencyFieldOpenExchangeTest, TestFunctionQuery, SOLR749Test, AlternateDirectoryTest, FileBasedSpellCheckerTest, BadCopyFieldTest, SpellPossibilityIteratorTest, SolrRequestParserTest, TestQueryUtils, TestRTGBase, DateFieldTest, TestRandomDVFaceting, MoreLikeThisHandlerTest, TestPropInject, TestFiltering, UpdateRequestProcessorFactoryTest, TestUpdate, HighlighterTest, TestPseudoReturnFields, TestUtils, SuggesterWFSTTest, TestMaxScoreQueryParser, SuggesterFSTTest, TestAddFieldRealTimeGet, TestCloudManagedSchemaAddField, TestReplicationHandler, StatelessScriptUpdateProcessorFactoryTest, TestAtomicUpdateErrorCases, SuggesterTSTTest, TestSurroundQueryParser, FieldMutatingUpdateProcessorTest, IndexSchemaTest, TestAnalyzedSuggestions, ScriptEngineTest, TestSearchPerf, OpenCloseCoreStressTest, TestSolrXMLSerializer, SolrCoreTest, MultiTermTest, NotRequiredUniqueKeyTest, DocumentAnalysisRequestHandlerTest, SolrPluginUtilsTest, TestFieldTypeCollectionResource, TestComponentsName, TestArbitraryIndexDir, AutoCommitTest, ShardRoutingCustomTest, TestDistributedGrouping, TestValueSourceCache, CachingDirectoryFactoryTest, DirectUpdateHandlerTest, TestWriterPerf, SliceStateUpdateTest, PreAnalyzedFieldTest, BasicFunctionalityTest, DocumentBuilderTest, ChaosMonkeyShardSplitTest]
[junit4:junit4] Completed in 85.10s, 1 test, 1 error <<< FAILURES!
[...truncated 719 lines...]
BUILD FAILED
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/build.xml:386: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/build.xml:366: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:181: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:437: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1243: The following error occurred while executing this line:
/Users/jenkins/jenkins-slave/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:887: There were test failures: 296 suites, 1234 tests, 1 error, 18 ignored (6 assumptions)
Total time: 118 minutes 43 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure