You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2013/05/31 00:52:03 UTC
[JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4029 - Still
Failing
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4029/
1 tests failed.
FAILED: org.apache.solr.cloud.ChaosMonkeyShardSplitTest.testDistribSearch
Error Message:
Server at http://127.0.0.1:27681 returned non ok status:500, message:Server Error
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:27681 returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([FFC84230815A5C2E:7E2ECC28F6053C12]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9463 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeyShardSplitTest
[junit4:junit4] 2> 598532 T1399 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 598539 T1399 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeyShardSplitTest-1369952983003
[junit4:junit4] 2> 598541 T1399 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 598542 T1400 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 598643 T1399 oasc.ZkTestServer.run start zk server on port:27674
[junit4:junit4] 2> 598644 T1399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 598648 T1406 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20e224d9 name:ZooKeeperConnection Watcher:127.0.0.1:27674 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 598648 T1399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 598649 T1399 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 598668 T1399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 598670 T1408 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@563f7584 name:ZooKeeperConnection Watcher:127.0.0.1:27674/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 598671 T1399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 598671 T1399 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 598676 T1399 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 598683 T1399 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 598686 T1399 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 598689 T1399 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 598689 T1399 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 598695 T1399 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 598696 T1399 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 598803 T1399 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 598804 T1399 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 598815 T1399 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 598816 T1399 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 598819 T1399 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 598820 T1399 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 598823 T1399 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 598824 T1399 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 598828 T1399 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 598829 T1399 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 598832 T1399 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 598832 T1399 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 598835 T1399 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 598836 T1399 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 599105 T1399 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 599110 T1399 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:27677
[junit4:junit4] 2> 599111 T1399 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 599111 T1399 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 599112 T1399 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369952983310
[junit4:junit4] 2> 599112 T1399 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369952983310/solr.xml
[junit4:junit4] 2> 599113 T1399 oasc.CoreContainer.<init> New CoreContainer 1225683015
[junit4:junit4] 2> 599114 T1399 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369952983310/'
[junit4:junit4] 2> 599114 T1399 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369952983310/'
[junit4:junit4] 2> 599242 T1399 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 599243 T1399 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 599244 T1399 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 599244 T1399 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 599245 T1399 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 599245 T1399 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 599246 T1399 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 599246 T1399 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 599247 T1399 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 599247 T1399 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 599266 T1399 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 599267 T1399 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:27674/solr
[junit4:junit4] 2> 599267 T1399 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 599268 T1399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 599271 T1419 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fe02cd3 name:ZooKeeperConnection Watcher:127.0.0.1:27674 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 599271 T1399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 599274 T1399 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 599290 T1399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 599295 T1421 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@372d8ede name:ZooKeeperConnection Watcher:127.0.0.1:27674/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 599295 T1399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 599341 T1399 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 599362 T1399 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 599372 T1399 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 599374 T1399 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:27677_
[junit4:junit4] 2> 599376 T1399 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:27677_
[junit4:junit4] 2> 599405 T1399 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 599523 T1399 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 599561 T1399 oasc.Overseer.start Overseer (id=89781238694739971-127.0.0.1:27677_-n_0000000000) starting
[junit4:junit4] 2> 599696 T1399 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 599701 T1423 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 599702 T1399 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 599766 T1399 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 599769 T1399 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 599771 T1422 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 599775 T1424 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369952983310/collection1
[junit4:junit4] 2> 599776 T1424 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 599776 T1424 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 599777 T1424 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 599778 T1424 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369952983310/collection1/'
[junit4:junit4] 2> 599780 T1424 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369952983310/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 599780 T1424 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369952983310/collection1/lib/README' to classloader
[junit4:junit4] 2> 599833 T1424 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 599937 T1424 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 600038 T1424 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 600044 T1424 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 600599 T1424 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 600603 T1424 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 600605 T1424 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 600611 T1424 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 600634 T1424 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 600635 T1424 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369952983310/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/control/data/
[junit4:junit4] 2> 600635 T1424 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b07bb1
[junit4:junit4] 2> 600636 T1424 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 600636 T1424 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/control/data
[junit4:junit4] 2> 600637 T1424 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/control/data/index/
[junit4:junit4] 2> 600637 T1424 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 600637 T1424 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/control/data/index
[junit4:junit4] 2> 600642 T1424 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@585583aa lockFactory=org.apache.lucene.store.NativeFSLockFactory@7637b764),segFN=segments_1,generation=1}
[junit4:junit4] 2> 600642 T1424 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 600645 T1424 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 600645 T1424 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 600646 T1424 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 600647 T1424 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 600647 T1424 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 600647 T1424 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 600648 T1424 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 600648 T1424 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 600649 T1424 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 600664 T1424 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 600671 T1424 oass.SolrIndexSearcher.<init> Opening Searcher@3cce377b main
[junit4:junit4] 2> 600671 T1424 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/control/data/tlog
[junit4:junit4] 2> 600672 T1424 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 600672 T1424 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 600676 T1425 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3cce377b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 600678 T1424 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 600678 T1424 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 601275 T1422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 601276 T1422 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:27677_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:27677"}
[junit4:junit4] 2> 601276 T1422 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 601277 T1422 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 601331 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 601681 T1424 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 601681 T1424 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:27677 collection:control_collection shard:shard1
[junit4:junit4] 2> 601682 T1424 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 601699 T1424 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 601710 T1424 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 601710 T1424 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 601710 T1424 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:27677/collection1/
[junit4:junit4] 2> 601711 T1424 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 601711 T1424 oasc.SyncStrategy.syncToMe http://127.0.0.1:27677/collection1/ has no replicas
[junit4:junit4] 2> 601711 T1424 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:27677/collection1/
[junit4:junit4] 2> 601712 T1424 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 602835 T1422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 602888 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 602950 T1424 oasc.ZkController.register We are http://127.0.0.1:27677/collection1/ and leader is http://127.0.0.1:27677/collection1/
[junit4:junit4] 2> 602950 T1424 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:27677
[junit4:junit4] 2> 602950 T1424 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 602950 T1424 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 602951 T1424 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 602953 T1424 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 602954 T1399 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 602955 T1399 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 602956 T1399 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 602972 T1399 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 602974 T1399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 602976 T1428 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d815bc4 name:ZooKeeperConnection Watcher:127.0.0.1:27674/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 602977 T1399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 602979 T1399 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 602981 T1399 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 603241 T1399 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 603244 T1399 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:27681
[junit4:junit4] 2> 603245 T1399 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 603245 T1399 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 603246 T1399 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369952987444
[junit4:junit4] 2> 603246 T1399 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369952987444/solr.xml
[junit4:junit4] 2> 603247 T1399 oasc.CoreContainer.<init> New CoreContainer 1561425223
[junit4:junit4] 2> 603248 T1399 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369952987444/'
[junit4:junit4] 2> 603248 T1399 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369952987444/'
[junit4:junit4] 2> 603376 T1399 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 603377 T1399 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 603377 T1399 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 603378 T1399 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 603378 T1399 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 603379 T1399 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 603379 T1399 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 603380 T1399 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 603380 T1399 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 603381 T1399 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 603404 T1399 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 603405 T1399 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:27674/solr
[junit4:junit4] 2> 603405 T1399 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 603406 T1399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 603409 T1439 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2244eac6 name:ZooKeeperConnection Watcher:127.0.0.1:27674 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 603410 T1399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 603412 T1399 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 603424 T1399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 603426 T1441 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3442aafa name:ZooKeeperConnection Watcher:127.0.0.1:27674/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 603426 T1399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 603437 T1399 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 604403 T1422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 604403 T1422 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:27677__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:27677_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:27677"}
[junit4:junit4] 2> 604456 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 604456 T1399 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:27681_
[junit4:junit4] 2> 604456 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 604456 T1428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 604466 T1399 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:27681_
[junit4:junit4] 2> 604468 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 604468 T1428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 604469 T1441 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 604469 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 604470 T1421 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 604470 T1428 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 604515 T1442 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369952987444/collection1
[junit4:junit4] 2> 604516 T1442 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 604516 T1442 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 604517 T1442 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 604518 T1442 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369952987444/collection1/'
[junit4:junit4] 2> 604520 T1442 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369952987444/collection1/lib/README' to classloader
[junit4:junit4] 2> 604520 T1442 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369952987444/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 604573 T1442 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 604634 T1442 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 604736 T1442 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 604742 T1442 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 605312 T1442 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 605316 T1442 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 605319 T1442 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 605324 T1442 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 605347 T1442 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 605348 T1442 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369952987444/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty1/
[junit4:junit4] 2> 605348 T1442 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b07bb1
[junit4:junit4] 2> 605349 T1442 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 605349 T1442 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty1
[junit4:junit4] 2> 605350 T1442 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty1/index/
[junit4:junit4] 2> 605350 T1442 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 605351 T1442 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty1/index
[junit4:junit4] 2> 605355 T1442 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@52270da5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3f4b99de),segFN=segments_1,generation=1}
[junit4:junit4] 2> 605355 T1442 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 605358 T1442 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 605359 T1442 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 605359 T1442 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 605360 T1442 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 605361 T1442 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 605361 T1442 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 605361 T1442 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 605362 T1442 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 605363 T1442 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 605379 T1442 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 605386 T1442 oass.SolrIndexSearcher.<init> Opening Searcher@56b84a61 main
[junit4:junit4] 2> 605387 T1442 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty1/tlog
[junit4:junit4] 2> 605388 T1442 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 605388 T1442 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 605393 T1443 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@56b84a61 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 605395 T1442 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 605395 T1442 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 605968 T1422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 605969 T1422 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:27681_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:27681"}
[junit4:junit4] 2> 605970 T1422 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 605970 T1422 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 606016 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 606016 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 606016 T1428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 606398 T1442 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 606398 T1442 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:27681 collection:collection1 shard:shard1
[junit4:junit4] 2> 606399 T1442 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 606416 T1442 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 606428 T1442 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 606428 T1442 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 606429 T1442 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:27681/collection1/
[junit4:junit4] 2> 606429 T1442 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 606429 T1442 oasc.SyncStrategy.syncToMe http://127.0.0.1:27681/collection1/ has no replicas
[junit4:junit4] 2> 606429 T1442 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:27681/collection1/
[junit4:junit4] 2> 606430 T1442 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 607520 T1422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 607538 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 607538 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 607538 T1428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 607558 T1442 oasc.ZkController.register We are http://127.0.0.1:27681/collection1/ and leader is http://127.0.0.1:27681/collection1/
[junit4:junit4] 2> 607558 T1442 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:27681
[junit4:junit4] 2> 607558 T1442 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 607558 T1442 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 607559 T1442 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 607560 T1442 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 607561 T1399 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 607562 T1399 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 607563 T1399 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 607839 T1399 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 607842 T1399 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:27688
[junit4:junit4] 2> 607843 T1399 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 607843 T1399 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 607844 T1399 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369952992043
[junit4:junit4] 2> 607844 T1399 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369952992043/solr.xml
[junit4:junit4] 2> 607845 T1399 oasc.CoreContainer.<init> New CoreContainer 468045223
[junit4:junit4] 2> 607846 T1399 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369952992043/'
[junit4:junit4] 2> 607846 T1399 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369952992043/'
[junit4:junit4] 2> 607973 T1399 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 607974 T1399 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 607974 T1399 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 607975 T1399 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 607975 T1399 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 607976 T1399 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 607977 T1399 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 607977 T1399 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 607978 T1399 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 607978 T1399 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 607996 T1399 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 607997 T1399 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:27674/solr
[junit4:junit4] 2> 607998 T1399 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 607999 T1399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 608002 T1455 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54c3afd7 name:ZooKeeperConnection Watcher:127.0.0.1:27674 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 608003 T1399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 608018 T1399 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 608031 T1399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 608033 T1457 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e1e49ae name:ZooKeeperConnection Watcher:127.0.0.1:27674/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 608033 T1399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 608039 T1399 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 609042 T1399 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:27688_
[junit4:junit4] 2> 609043 T1422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 609044 T1399 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:27688_
[junit4:junit4] 2> 609045 T1422 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:27681__collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:27681_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:27681"}
[junit4:junit4] 2> 609058 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 609058 T1428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 609059 T1441 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 609059 T1457 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 609059 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 609066 T1457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 609066 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 609067 T1428 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 609067 T1428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 609076 T1421 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 609077 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 609079 T1458 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369952992043/collection1
[junit4:junit4] 2> 609080 T1458 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 609080 T1458 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 609081 T1458 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 609082 T1458 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369952992043/collection1/'
[junit4:junit4] 2> 609084 T1458 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369952992043/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 609084 T1458 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369952992043/collection1/lib/README' to classloader
[junit4:junit4] 2> 609136 T1458 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 609196 T1458 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 609297 T1458 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 609303 T1458 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 609874 T1458 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 609879 T1458 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 609881 T1458 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 609886 T1458 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 609910 T1458 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 609910 T1458 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369952992043/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty2/
[junit4:junit4] 2> 609910 T1458 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b07bb1
[junit4:junit4] 2> 609911 T1458 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 609912 T1458 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty2
[junit4:junit4] 2> 609913 T1458 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty2/index/
[junit4:junit4] 2> 609913 T1458 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 609914 T1458 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty2/index
[junit4:junit4] 2> 609917 T1458 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@c6fc1ab lockFactory=org.apache.lucene.store.NativeFSLockFactory@3b7cb7c5),segFN=segments_1,generation=1}
[junit4:junit4] 2> 609918 T1458 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 609921 T1458 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 609921 T1458 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 609922 T1458 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 609923 T1458 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 609923 T1458 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 609924 T1458 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 609924 T1458 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 609925 T1458 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 609925 T1458 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 609942 T1458 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 609949 T1458 oass.SolrIndexSearcher.<init> Opening Searcher@38941e51 main
[junit4:junit4] 2> 609949 T1458 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty2/tlog
[junit4:junit4] 2> 609951 T1458 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 609951 T1458 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 609956 T1459 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@38941e51 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 609958 T1458 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 609958 T1458 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 610577 T1422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 610578 T1422 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:27688_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:27688"}
[junit4:junit4] 2> 610578 T1422 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 610579 T1422 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 610619 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 610620 T1457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 610620 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 610620 T1428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 610961 T1458 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 610961 T1458 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:27688 collection:collection1 shard:shard2
[junit4:junit4] 2> 610962 T1458 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 610978 T1458 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 610981 T1458 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 610981 T1458 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 610981 T1458 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:27688/collection1/
[junit4:junit4] 2> 610982 T1458 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 610982 T1458 oasc.SyncStrategy.syncToMe http://127.0.0.1:27688/collection1/ has no replicas
[junit4:junit4] 2> 610982 T1458 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:27688/collection1/
[junit4:junit4] 2> 610983 T1458 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 612124 T1422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 612144 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 612144 T1457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 612144 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 612144 T1428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 612161 T1458 oasc.ZkController.register We are http://127.0.0.1:27688/collection1/ and leader is http://127.0.0.1:27688/collection1/
[junit4:junit4] 2> 612161 T1458 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:27688
[junit4:junit4] 2> 612161 T1458 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 612161 T1458 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 612162 T1458 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 612163 T1458 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 612165 T1399 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 612165 T1399 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 612166 T1399 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 612443 T1399 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 612447 T1399 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:31637
[junit4:junit4] 2> 612448 T1399 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 612448 T1399 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 612449 T1399 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369952996646
[junit4:junit4] 2> 612449 T1399 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369952996646/solr.xml
[junit4:junit4] 2> 612450 T1399 oasc.CoreContainer.<init> New CoreContainer 2058838736
[junit4:junit4] 2> 612451 T1399 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369952996646/'
[junit4:junit4] 2> 612451 T1399 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369952996646/'
[junit4:junit4] 2> 612578 T1399 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 612579 T1399 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 612579 T1399 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 612580 T1399 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 612580 T1399 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 612581 T1399 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 612582 T1399 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 612582 T1399 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 612583 T1399 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 612583 T1399 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 612600 T1399 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 612601 T1399 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:27674/solr
[junit4:junit4] 2> 612601 T1399 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 612602 T1399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 612605 T1471 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5de9e398 name:ZooKeeperConnection Watcher:127.0.0.1:27674 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 612606 T1399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 612622 T1399 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 612634 T1399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 612636 T1473 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63ac56f name:ZooKeeperConnection Watcher:127.0.0.1:27674/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 612637 T1399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 612642 T1399 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 613645 T1399 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:31637_
[junit4:junit4] 2> 613647 T1399 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:31637_
[junit4:junit4] 2> 613650 T1428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 613650 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 613651 T1457 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 613651 T1441 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 613651 T1473 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 613651 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 613651 T1457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 613652 T1428 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 613652 T1422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 613653 T1422 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:27688__collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:27688_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:27688"}
[junit4:junit4] 2> 613658 T1474 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369952996646/collection1
[junit4:junit4] 2> 613659 T1474 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 613665 T1474 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 613665 T1474 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 613666 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 613666 T1428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 613666 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 613666 T1457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 613668 T1474 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369952996646/collection1/'
[junit4:junit4] 2> 613669 T1474 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369952996646/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 613670 T1474 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369952996646/collection1/lib/README' to classloader
[junit4:junit4] 2> 613670 T1421 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 613670 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 613721 T1474 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 613781 T1474 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 613883 T1474 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 613888 T1474 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 614464 T1474 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 614468 T1474 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 614471 T1474 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 614477 T1474 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 614501 T1474 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 614502 T1474 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369952996646/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty3/
[junit4:junit4] 2> 614502 T1474 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b07bb1
[junit4:junit4] 2> 614503 T1474 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 614503 T1474 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty3
[junit4:junit4] 2> 614504 T1474 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty3/index/
[junit4:junit4] 2> 614504 T1474 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 614505 T1474 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty3/index
[junit4:junit4] 2> 614509 T1474 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@769e02c0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6aa0ce6),segFN=segments_1,generation=1}
[junit4:junit4] 2> 614509 T1474 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 614512 T1474 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 614512 T1474 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 614513 T1474 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 614514 T1474 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 614515 T1474 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 614515 T1474 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 614515 T1474 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 614516 T1474 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 614516 T1474 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 614533 T1474 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 614540 T1474 oass.SolrIndexSearcher.<init> Opening Searcher@90f626b main
[junit4:junit4] 2> 614541 T1474 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty3/tlog
[junit4:junit4] 2> 614542 T1474 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 614542 T1474 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 614546 T1475 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@90f626b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 614549 T1474 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 614549 T1474 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 615171 T1422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 615172 T1422 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:31637_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:31637"}
[junit4:junit4] 2> 615173 T1422 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 615173 T1422 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 615203 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 615203 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 615203 T1457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 615203 T1428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 615203 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 615552 T1474 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 615552 T1474 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:31637 collection:collection1 shard:shard1
[junit4:junit4] 2> 615555 T1474 oasc.ZkController.register We are http://127.0.0.1:31637/collection1/ and leader is http://127.0.0.1:27681/collection1/
[junit4:junit4] 2> 615555 T1474 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:31637
[junit4:junit4] 2> 615556 T1474 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 615556 T1474 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C426 name=collection1 org.apache.solr.core.SolrCore@7ffa89a url=http://127.0.0.1:31637/collection1 node=127.0.0.1:31637_ C426_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:31637_, base_url=http://127.0.0.1:31637}
[junit4:junit4] 2> 615556 T1476 C426 P31637 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 615557 T1474 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 615557 T1476 C426 P31637 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 615558 T1476 C426 P31637 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 615558 T1476 C426 P31637 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 615558 T1399 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 615559 T1399 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 615559 T1399 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 615559 T1476 C426 P31637 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 615585 T1431 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:31637__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 615854 T1399 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 615857 T1399 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:40188
[junit4:junit4] 2> 615858 T1399 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 615859 T1399 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 615859 T1399 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369953000047
[junit4:junit4] 2> 615860 T1399 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369953000047/solr.xml
[junit4:junit4] 2> 615860 T1399 oasc.CoreContainer.<init> New CoreContainer 805350865
[junit4:junit4] 2> 615861 T1399 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369953000047/'
[junit4:junit4] 2> 615862 T1399 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369953000047/'
[junit4:junit4] 2> 615991 T1399 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 615992 T1399 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 615993 T1399 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 615993 T1399 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 615994 T1399 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 615994 T1399 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 615995 T1399 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 615995 T1399 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 615996 T1399 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 615997 T1399 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 616014 T1399 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 616015 T1399 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:27674/solr
[junit4:junit4] 2> 616015 T1399 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 616017 T1399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 616020 T1488 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6da5b5f0 name:ZooKeeperConnection Watcher:127.0.0.1:27674 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 616021 T1399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 616036 T1399 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 616048 T1399 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 616051 T1490 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e508d8b name:ZooKeeperConnection Watcher:127.0.0.1:27674/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 616051 T1399 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 616062 T1399 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 616708 T1422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 616709 T1422 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:31637__collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:31637_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:31637"}
[junit4:junit4] 2> 616723 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 616723 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 616723 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 616723 T1457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 616723 T1428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 616723 T1490 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 617066 T1399 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40188_
[junit4:junit4] 2> 617068 T1399 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40188_
[junit4:junit4] 2> 617070 T1490 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 617071 T1428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 617071 T1457 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 617071 T1441 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 617071 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 617071 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 617072 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 617071 T1457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 617073 T1428 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 617072 T1490 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 617081 T1473 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 617082 T1421 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 617086 T1491 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369953000047/collection1
[junit4:junit4] 2> 617087 T1491 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 617087 T1491 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 617088 T1491 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 617089 T1491 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369953000047/collection1/'
[junit4:junit4] 2> 617091 T1491 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369953000047/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 617091 T1491 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369953000047/collection1/lib/README' to classloader
[junit4:junit4] 2> 617143 T1491 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 617203 T1491 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 617305 T1491 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 617311 T1491 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 617588 T1431 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:31637__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 617588 T1431 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:31637__collection1&state=recovering&nodeName=127.0.0.1:31637_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2003
[junit4:junit4] 2> 617884 T1491 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 617888 T1491 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 617891 T1491 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 617896 T1491 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 617919 T1491 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 617919 T1491 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369953000047/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty4/
[junit4:junit4] 2> 617919 T1491 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b07bb1
[junit4:junit4] 2> 617920 T1491 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 617921 T1491 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty4
[junit4:junit4] 2> 617921 T1491 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty4/index/
[junit4:junit4] 2> 617921 T1491 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 617923 T1491 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty4/index
[junit4:junit4] 2> 617926 T1491 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7b39a835 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2be241ea),segFN=segments_1,generation=1}
[junit4:junit4] 2> 617927 T1491 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 617930 T1491 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 617930 T1491 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 617931 T1491 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 617932 T1491 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 617932 T1491 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 617933 T1491 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 617933 T1491 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 617934 T1491 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 617934 T1491 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 617950 T1491 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 617958 T1491 oass.SolrIndexSearcher.<init> Opening Searcher@5d84e375 main
[junit4:junit4] 2> 617958 T1491 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty4/tlog
[junit4:junit4] 2> 617959 T1491 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 617960 T1491 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 617964 T1492 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5d84e375 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 617967 T1491 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 617967 T1491 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 618228 T1422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 618229 T1422 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40188_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40188"}
[junit4:junit4] 2> 618229 T1422 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 618229 T1422 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 618243 T1490 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 618243 T1457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 618243 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 618243 T1428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 618243 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 618243 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 618970 T1491 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 618970 T1491 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40188 collection:collection1 shard:shard2
[junit4:junit4] 2> 618973 T1491 oasc.ZkController.register We are http://127.0.0.1:40188/collection1/ and leader is http://127.0.0.1:27688/collection1/
[junit4:junit4] 2> 618974 T1491 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40188
[junit4:junit4] 2> 618974 T1491 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 618974 T1491 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C427 name=collection1 org.apache.solr.core.SolrCore@3e20666e url=http://127.0.0.1:40188/collection1 node=127.0.0.1:40188_ C427_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:40188_, base_url=http://127.0.0.1:40188}
[junit4:junit4] 2> 618975 T1493 C427 P40188 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 618975 T1491 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 618975 T1493 C427 P40188 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 618976 T1493 C427 P40188 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 618976 T1493 C427 P40188 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 618977 T1399 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 618977 T1399 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 618978 T1399 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 618978 T1493 C427 P40188 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 619003 T1399 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 619004 T1447 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:40188__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 619004 T1399 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 619005 T1399 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 619006 T1399 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C428 name=collection1 org.apache.solr.core.SolrCore@7ffa89a url=http://127.0.0.1:31637/collection1 node=127.0.0.1:31637_ C428_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:31637_, base_url=http://127.0.0.1:31637}
[junit4:junit4] 2> 619590 T1476 C428 P31637 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:27681/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 619590 T1476 C428 P31637 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:31637 START replicas=[http://127.0.0.1:27681/collection1/] nUpdates=100
[junit4:junit4] 2> 619591 T1476 C428 P31637 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 619592 T1476 C428 P31637 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 619592 T1476 C428 P31637 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 619592 T1476 C428 P31637 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 619592 T1476 C428 P31637 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 619592 T1476 C428 P31637 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:27681/collection1/. core=collection1
[junit4:junit4] 2> 619593 T1476 C428 P31637 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C429 name=collection1 org.apache.solr.core.SolrCore@7536d0e7 url=http://127.0.0.1:27681/collection1 node=127.0.0.1:27681_ C429_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:27681_, base_url=http://127.0.0.1:27681, leader=true}
[junit4:junit4] 2> 619601 T1432 C429 P27681 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 619607 T1433 C429 P27681 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 619610 T1433 C429 P27681 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@52270da5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3f4b99de),segFN=segments_1,generation=1}
[junit4:junit4] 2> 619610 T1433 C429 P27681 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 619611 T1433 C429 P27681 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@52270da5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3f4b99de),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@52270da5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3f4b99de),segFN=segments_2,generation=2}
[junit4:junit4] 2> 619611 T1433 C429 P27681 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 619612 T1433 C429 P27681 oass.SolrIndexSearcher.<init> Opening Searcher@49006f27 realtime
[junit4:junit4] 2> 619612 T1433 C429 P27681 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 619613 T1433 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 619614 T1476 C428 P31637 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 619614 T1476 C428 P31637 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 619616 T1434 C429 P27681 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 619617 T1434 C429 P27681 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 619618 T1476 C428 P31637 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 619618 T1476 C428 P31637 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 619618 T1476 C428 P31637 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 619620 T1434 C429 P27681 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 619621 T1476 C428 P31637 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 619622 T1476 C428 P31637 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty3/index.20130531033004084
[junit4:junit4] 2> 619622 T1476 C428 P31637 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d213caa lockFactory=org.apache.lucene.store.NativeFSLockFactory@62012017) fullCopy=false
[junit4:junit4] 2> 619625 T1434 C429 P27681 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 619626 T1476 C428 P31637 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 619627 T1476 C428 P31637 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 619628 T1476 C428 P31637 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 619630 T1476 C428 P31637 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@769e02c0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6aa0ce6),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@769e02c0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6aa0ce6),segFN=segments_2,generation=2}
[junit4:junit4] 2> 619631 T1476 C428 P31637 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 619631 T1476 C428 P31637 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 619632 T1476 C428 P31637 oass.SolrIndexSearcher.<init> Opening Searcher@57898e4d main
[junit4:junit4] 2> 619633 T1475 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@57898e4d main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 619633 T1476 C428 P31637 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty3/index.20130531033004084 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty3/index.20130531033004084;done=true>>]
[junit4:junit4] 2> 619634 T1476 C428 P31637 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty3/index.20130531033004084
[junit4:junit4] 2> 619634 T1476 C428 P31637 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty3/index.20130531033004084
[junit4:junit4] 2> 619634 T1476 C428 P31637 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 619634 T1476 C428 P31637 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 619635 T1476 C428 P31637 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 619635 T1476 C428 P31637 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 619643 T1476 C428 P31637 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 619747 T1422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 619748 T1422 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:40188__collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40188_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40188"}
[junit4:junit4] 2> 619761 T1422 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:31637__collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:31637_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:31637"}
[junit4:junit4] 2> 619780 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 619780 T1490 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 619780 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 619780 T1457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 619780 T1428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 619780 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 620005 T1447 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:40188__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 620005 T1447 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:40188__collection1&state=recovering&nodeName=127.0.0.1:40188_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 620008 T1399 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 621010 T1399 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C427_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:40188_, base_url=http://127.0.0.1:40188}
[junit4:junit4] 2> 622007 T1493 C427 P40188 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:27688/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 622007 T1493 C427 P40188 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:40188 START replicas=[http://127.0.0.1:27688/collection1/] nUpdates=100
[junit4:junit4] 2> 622008 T1493 C427 P40188 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 622008 T1493 C427 P40188 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 622008 T1493 C427 P40188 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 622008 T1493 C427 P40188 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 622009 T1493 C427 P40188 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 622009 T1493 C427 P40188 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:27688/collection1/. core=collection1
[junit4:junit4] 2> 622010 T1493 C427 P40188 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 622013 T1399 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C430 name=collection1 org.apache.solr.core.SolrCore@a9adf0c url=http://127.0.0.1:27688/collection1 node=127.0.0.1:27688_ C430_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:27688_, base_url=http://127.0.0.1:27688, leader=true}
[junit4:junit4] 2> 622017 T1448 C430 P27688 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 622023 T1449 C430 P27688 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 622026 T1449 C430 P27688 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@c6fc1ab lockFactory=org.apache.lucene.store.NativeFSLockFactory@3b7cb7c5),segFN=segments_1,generation=1}
[junit4:junit4] 2> 622026 T1449 C430 P27688 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 622027 T1449 C430 P27688 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@c6fc1ab lockFactory=org.apache.lucene.store.NativeFSLockFactory@3b7cb7c5),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@c6fc1ab lockFactory=org.apache.lucene.store.NativeFSLockFactory@3b7cb7c5),segFN=segments_2,generation=2}
[junit4:junit4] 2> 622028 T1449 C430 P27688 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 622028 T1449 C430 P27688 oass.SolrIndexSearcher.<init> Opening Searcher@46e4d602 realtime
[junit4:junit4] 2> 622029 T1449 C430 P27688 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 622029 T1449 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 622030 T1493 C427 P40188 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 622030 T1493 C427 P40188 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 622032 T1450 C430 P27688 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 622032 T1450 C430 P27688 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 622033 T1493 C427 P40188 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 622033 T1493 C427 P40188 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 622033 T1493 C427 P40188 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 622036 T1450 C430 P27688 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 622037 T1493 C427 P40188 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 622038 T1493 C427 P40188 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty4/index.20130531033006500
[junit4:junit4] 2> 622038 T1493 C427 P40188 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d297bc5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2fb7fd24) fullCopy=false
[junit4:junit4] 2> 622041 T1450 C430 P27688 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 622042 T1493 C427 P40188 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 622043 T1493 C427 P40188 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 622043 T1493 C427 P40188 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 622045 T1493 C427 P40188 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7b39a835 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2be241ea),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7b39a835 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2be241ea),segFN=segments_2,generation=2}
[junit4:junit4] 2> 622046 T1493 C427 P40188 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 622046 T1493 C427 P40188 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 622046 T1493 C427 P40188 oass.SolrIndexSearcher.<init> Opening Searcher@2f10e660 main
[junit4:junit4] 2> 622047 T1492 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2f10e660 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 622048 T1493 C427 P40188 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty4/index.20130531033006500 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty4/index.20130531033006500;done=true>>]
[junit4:junit4] 2> 622048 T1493 C427 P40188 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty4/index.20130531033006500
[junit4:junit4] 2> 622048 T1493 C427 P40188 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty4/index.20130531033006500
[junit4:junit4] 2> 622049 T1493 C427 P40188 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 622049 T1493 C427 P40188 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 622049 T1493 C427 P40188 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 622049 T1493 C427 P40188 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 622051 T1493 C427 P40188 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 622793 T1422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 622794 T1422 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:40188__collection1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40188_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40188"}
[junit4:junit4] 2> 622800 T1473 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 622800 T1441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 622800 T1457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 622800 T1428 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 622800 T1490 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 622800 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 623015 T1399 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 623016 T1399 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C431 name=collection1 org.apache.solr.core.SolrCore@49653c03 url=http://127.0.0.1:27677/collection1 node=127.0.0.1:27677_ C431_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:27677_, base_url=http://127.0.0.1:27677, leader=true}
[junit4:junit4] 2> 623025 T1411 C431 P27677 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 623029 T1411 C431 P27677 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@585583aa lockFactory=org.apache.lucene.store.NativeFSLockFactory@7637b764),segFN=segments_1,generation=1}
[junit4:junit4] 2> 623030 T1411 C431 P27677 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 623030 T1411 C431 P27677 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@585583aa lockFactory=org.apache.lucene.store.NativeFSLockFactory@7637b764),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@585583aa lockFactory=org.apache.lucene.store.NativeFSLockFactory@7637b764),segFN=segments_2,generation=2}
[junit4:junit4] 2> 623031 T1411 C431 P27677 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 623031 T1411 C431 P27677 oass.SolrIndexSearcher.<init> Opening Searcher@34200f0f main
[junit4:junit4] 2> 623032 T1411 C431 P27677 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 623033 T1425 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@34200f0f main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 623034 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4] 2> 623047 T1435 C429 P27681 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 623048 T1435 C429 P27681 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@52270da5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3f4b99de),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@52270da5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3f4b99de),segFN=segments_3,generation=3}
[junit4:junit4] 2> 623048 T1435 C429 P27681 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 623049 T1435 C429 P27681 oass.SolrIndexSearcher.<init> Opening Searcher@7d355c56 main
[junit4:junit4] 2> 623050 T1435 C429 P27681 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 623050 T1443 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7d355c56 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 623051 T1435 C429 P27681 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:31637/collection1/, StdNode: http://127.0.0.1:27688/collection1/, StdNode: http://127.0.0.1:40188/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 623054 T1451 C430 P27688 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 623055 T1451 C430 P27688 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@c6fc1ab lockFactory=org.apache.lucene.store.NativeFSLockFactory@3b7cb7c5),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@c6fc1ab lockFactory=org.apache.lucene.store.NativeFSLockFactory@3b7cb7c5),segFN=segments_3,generation=3}
[junit4:junit4] 2> 623056 T1451 C430 P27688 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 623056 T1451 C430 P27688 oass.SolrIndexSearcher.<init> Opening Searcher@2e430119 main
[junit4:junit4] 2> 623057 T1451 C430 P27688 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 623057 T1459 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2e430119 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 623058 T1451 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4] 2> C428_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:31637_, base_url=http://127.0.0.1:31637}
[junit4:junit4] 2> 623061 T1463 C428 P31637 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> C427_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:40188_, base_url=http://127.0.0.1:40188}
[junit4:junit4] 2> 623061 T1480 C427 P40188 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 623062 T1463 C428 P31637 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@769e02c0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6aa0ce6),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@769e02c0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6aa0ce6),segFN=segments_3,generation=3}
[junit4:junit4] 2> 623063 T1480 C427 P40188 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7b39a835 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2be241ea),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7b39a835 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2be241ea),segFN=segments_3,generation=3}
[junit4:junit4] 2> 623063 T1463 C428 P31637 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 623064 T1480 C427 P40188 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 623064 T1463 C428 P31637 oass.SolrIndexSearcher.<init> Opening Searcher@2ab19717 main
[junit4:junit4] 2> 623065 T1480 C427 P40188 oass.SolrIndexSearcher.<init> Opening Searcher@43bffb5d main
[junit4:junit4] 2> 623065 T1463 C428 P31637 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 623066 T1475 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2ab19717 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 623066 T1480 C427 P40188 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 623066 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 623066 T1492 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@43bffb5d main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 623067 T1480 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 623068 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 21
[junit4:junit4] 2> 623069 T1399 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 623072 T1436 C429 P27681 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 623074 T1464 C428 P31637 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 623076 T1452 C430 P27688 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 623078 T1481 C427 P40188 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 625084 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1436499846937706496)} 0 2
[junit4:junit4] 2> 625093 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1436499846941900800&update.from=http://127.0.0.1:27688/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1436499846941900800)} 0 2
[junit4:junit4] 2> 625097 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1436499846946095104&update.from=http://127.0.0.1:27681/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1436499846946095104)} 0 2
[junit4:junit4] 2> 625097 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1436499846946095104)} 0 6
[junit4:junit4] 2> 625098 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1436499846941900800)} 0 11
[junit4:junit4] 2> 625102 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1436499846956580864)]} 0 1
[junit4:junit4] 2> 625111 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1436499846962872320)]} 0 1
[junit4:junit4] 2> 625111 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1436499846962872320)]} 0 4
[junit4:junit4] 2> 625112 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0]} 0 7
[junit4:junit4] 2> 625115 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1436499846971260928)]} 0 0
[junit4:junit4] 2> 625123 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1436499846976503808)]} 0 0
[junit4:junit4] 2> 625124 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1436499846976503808)]} 0 4
[junit4:junit4] 2> 625125 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1]} 0 7
[junit4:junit4] 2> 625128 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1436499846984892416)]} 0 0
[junit4:junit4] 2> 625135 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1436499846988038144)]} 0 1
[junit4:junit4] 2> 625135 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1436499846988038144)]} 0 4
[junit4:junit4] 2> 625138 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1436499846995378176)]} 0 0
[junit4:junit4] 2> 625144 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[3 (1436499846998523904)]} 0 0
[junit4:junit4] 2> 625145 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1436499846998523904)]} 0 4
[junit4:junit4] 2> 625148 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1436499847005863936)]} 0 0
[junit4:junit4] 2> 625156 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[4 (1436499847011106816)]} 0 0
[junit4:junit4] 2> 625157 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[4 (1436499847011106816)]} 0 4
[junit4:junit4] 2> 625158 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4]} 0 7
[junit4:junit4] 2> 625160 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1436499847018446848)]} 0 0
[junit4:junit4] 2> 625168 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[5 (1436499847023689728)]} 0 0
[junit4:junit4] 2> 625169 T1451 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[5 (1436499847023689728)]} 0 4
[junit4:junit4] 2> 625170 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5]} 0 7
[junit4:junit4] 2> 625172 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1436499847031029760)]} 0 0
[junit4:junit4] 2> 625180 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[6 (1436499847036272640)]} 0 0
[junit4:junit4] 2> 625181 T1451 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[6 (1436499847036272640)]} 0 4
[junit4:junit4] 2> 625182 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6]} 0 7
[junit4:junit4] 2> 625184 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1436499847043612672)]} 0 0
[junit4:junit4] 2> 625192 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[7 (1436499847048855552)]} 0 0
[junit4:junit4] 2> 625193 T1451 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[7 (1436499847048855552)]} 0 4
[junit4:junit4] 2> 625194 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7]} 0 7
[junit4:junit4] 2> 625196 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1436499847056195584)]} 0 0
[junit4:junit4] 2> 625202 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[8 (1436499847059341312)]} 0 0
[junit4:junit4] 2> 625203 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1436499847059341312)]} 0 4
[junit4:junit4] 2> 625206 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1436499847066681344)]} 0 0
[junit4:junit4] 2> 625214 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[9 (1436499847071924224)]} 0 0
[junit4:junit4] 2> 625215 T1451 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[9 (1436499847071924224)]} 0 4
[junit4:junit4] 2> 625216 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9]} 0 7
[junit4:junit4] 2> 625218 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1436499847079264256)]} 0 0
[junit4:junit4] 2> 625224 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10 (1436499847082409984)]} 0 0
[junit4:junit4] 2> 625225 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1436499847082409984)]} 0 4
[junit4:junit4] 2> 625228 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1436499847089750016)]} 0 0
[junit4:junit4] 2> 625236 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[11 (1436499847094992896)]} 0 0
[junit4:junit4] 2> 625237 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[11 (1436499847094992896)]} 0 4
[junit4:junit4] 2> 625238 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11]} 0 7
[junit4:junit4] 2> 625241 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12 (1436499847103381504)]} 0 0
[junit4:junit4] 2> 625249 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1436499847108624384)]} 0 0
[junit4:junit4] 2> 625250 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[12 (1436499847108624384)]} 0 4
[junit4:junit4] 2> 625251 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12]} 0 7
[junit4:junit4] 2> 625253 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13 (1436499847115964416)]} 0 0
[junit4:junit4] 2> 625259 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1436499847119110144)]} 0 0
[junit4:junit4] 2> 625260 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13 (1436499847119110144)]} 0 4
[junit4:junit4] 2> 625263 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14 (1436499847126450176)]} 0 0
[junit4:junit4] 2> 625269 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1436499847129595904)]} 0 0
[junit4:junit4] 2> 625270 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14 (1436499847129595904)]} 0 4
[junit4:junit4] 2> 625273 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1436499847136935936)]} 0 0
[junit4:junit4] 2> 625281 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1436499847142178816)]} 0 0
[junit4:junit4] 2> 625282 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[15 (1436499847142178816)]} 0 4
[junit4:junit4] 2> 625283 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15]} 0 7
[junit4:junit4] 2> 625285 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1436499847149518848)]} 0 0
[junit4:junit4] 2> 625291 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1436499847152664576)]} 0 0
[junit4:junit4] 2> 625292 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1436499847152664576)]} 0 4
[junit4:junit4] 2> 625295 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1436499847160004608)]} 0 0
[junit4:junit4] 2> 625301 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1436499847163150336)]} 0 0
[junit4:junit4] 2> 625302 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1436499847163150336)]} 0 4
[junit4:junit4] 2> 625305 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18 (1436499847170490368)]} 0 0
[junit4:junit4] 2> 625313 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1436499847175733248)]} 0 0
[junit4:junit4] 2> 625314 T1451 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[18 (1436499847175733248)]} 0 4
[junit4:junit4] 2> 625315 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18]} 0 7
[junit4:junit4] 2> 625317 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1436499847183073280)]} 0 0
[junit4:junit4] 2> 625323 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1436499847186219008)]} 0 0
[junit4:junit4] 2> 625324 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1436499847186219008)]} 0 4
[junit4:junit4] 2> 625327 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1436499847193559040)]} 0 0
[junit4:junit4] 2> 625333 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1436499847196704768)]} 0 0
[junit4:junit4] 2> 625334 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1436499847196704768)]} 0 4
[junit4:junit4] 2> 625337 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1436499847204044800)]} 0 0
[junit4:junit4] 2> 625343 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1436499847207190528)]} 0 0
[junit4:junit4] 2> 625344 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1436499847207190528)]} 0 4
[junit4:junit4] 2> 625347 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[22 (1436499847214530560)]} 0 0
[junit4:junit4] 2> 625353 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1436499847217676288)]} 0 0
[junit4:junit4] 2> 625354 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[22 (1436499847217676288)]} 0 4
[junit4:junit4] 2> 625357 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[23 (1436499847225016320)]} 0 0
[junit4:junit4] 2> 625363 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1436499847228162048)]} 0 0
[junit4:junit4] 2> 625364 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[23 (1436499847228162048)]} 0 4
[junit4:junit4] 2> 625367 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[24 (1436499847235502080)]} 0 0
[junit4:junit4] 2> 625375 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1436499847240744960)]} 0 0
[junit4:junit4] 2> 625376 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[24 (1436499847240744960)]} 0 4
[junit4:junit4] 2> 625377 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[24]} 0 7
[junit4:junit4] 2> 625379 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[25 (1436499847248084992)]} 0 0
[junit4:junit4] 2> 625387 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1436499847253327872)]} 0 0
[junit4:junit4] 2> 625388 T1451 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[25 (1436499847253327872)]} 0 4
[junit4:junit4] 2> 625389 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[25]} 0 7
[junit4:junit4] 2> 625391 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[26 (1436499847260667904)]} 0 0
[junit4:junit4] 2> 625397 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1436499847263813632)]} 0 0
[junit4:junit4] 2> 625398 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[26 (1436499847263813632)]} 0 4
[junit4:junit4] 2> 625401 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[27 (1436499847271153664)]} 0 0
[junit4:junit4] 2> 625409 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1436499847276396544)]} 0 0
[junit4:junit4] 2> 625410 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[27 (1436499847276396544)]} 0 4
[junit4:junit4] 2> 625411 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[27]} 0 7
[junit4:junit4] 2> 625415 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[28 (1436499847284785152)]} 0 1
[junit4:junit4] 2> 625423 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1436499847291076608)]} 0 0
[junit4:junit4] 2> 625424 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[28 (1436499847291076608)]} 0 4
[junit4:junit4] 2> 625425 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[28]} 0 7
[junit4:junit4] 2> 625427 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[29 (1436499847298416640)]} 0 0
[junit4:junit4] 2> 625436 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1436499847304708096)]} 0 0
[junit4:junit4] 2> 625437 T1451 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[29 (1436499847304708096)]} 0 4
[junit4:junit4] 2> 625438 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[29]} 0 8
[junit4:junit4] 2> 625442 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[30 (1436499847313096704)]} 0 1
[junit4:junit4] 2> 625449 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1436499847318339584)]} 0 0
[junit4:junit4] 2> 625450 T1451 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[30 (1436499847318339584)]} 0 4
[junit4:junit4] 2> 625451 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[30]} 0 7
[junit4:junit4] 2> 625453 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[31 (1436499847325679616)]} 0 0
[junit4:junit4] 2> 625461 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1436499847330922496)]} 0 0
[junit4:junit4] 2> 625462 T1451 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[31 (1436499847330922496)]} 0 4
[junit4:junit4] 2> 625463 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[31]} 0 7
[junit4:junit4] 2> 625465 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[32 (1436499847338262528)]} 0 0
[junit4:junit4] 2> 625471 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1436499847341408256)]} 0 0
[junit4:junit4] 2> 625472 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[32 (1436499847341408256)]} 0 4
[junit4:junit4] 2> 625475 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[33 (1436499847348748288)]} 0 0
[junit4:junit4] 2> 625483 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1436499847353991168)]} 0 0
[junit4:junit4] 2> 625484 T1451 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[33 (1436499847353991168)]} 0 4
[junit4:junit4] 2> 625485 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[33]} 0 7
[junit4:junit4] 2> 625487 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[34 (1436499847361331200)]} 0 0
[junit4:junit4] 2> 625493 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1436499847364476928)]} 0 0
[junit4:junit4] 2> 625494 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[34 (1436499847364476928)]} 0 4
[junit4:junit4] 2> 625497 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[35 (1436499847371816960)]} 0 0
[junit4:junit4] 2> 625503 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1436499847374962688)]} 0 0
[junit4:junit4] 2> 625504 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[35 (1436499847374962688)]} 0 4
[junit4:junit4] 2> 625507 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[36 (1436499847382302720)]} 0 0
[junit4:junit4] 2> 625513 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1436499847385448448)]} 0 0
[junit4:junit4] 2> 625514 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[36 (1436499847385448448)]} 0 4
[junit4:junit4] 2> 625517 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[37 (1436499847392788480)]} 0 0
[junit4:junit4] 2> 625523 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1436499847395934208)]} 0 0
[junit4:junit4] 2> 625524 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[37 (1436499847395934208)]} 0 4
[junit4:junit4] 2> 625527 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[38 (1436499847403274240)]} 0 0
[junit4:junit4] 2> 625533 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[38 (1436499847406419968)]} 0 0
[junit4:junit4] 2> 625534 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[38 (1436499847406419968)]} 0 4
[junit4:junit4] 2> 625537 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[39 (1436499847413760000)]} 0 0
[junit4:junit4] 2> 625543 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[39 (1436499847416905728)]} 0 0
[junit4:junit4] 2> 625544 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[39 (1436499847416905728)]} 0 4
[junit4:junit4] 2> 625547 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[40 (1436499847424245760)]} 0 0
[junit4:junit4] 2> 625555 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[40 (1436499847429488640)]} 0 0
[junit4:junit4] 2> 625556 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[40 (1436499847429488640)]} 0 4
[junit4:junit4] 2> 625557 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[40]} 0 7
[junit4:junit4] 2> 625559 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[41 (1436499847436828672)]} 0 0
[junit4:junit4] 2> 625565 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[41 (1436499847439974400)]} 0 0
[junit4:junit4] 2> 625566 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[41 (1436499847439974400)]} 0 4
[junit4:junit4] 2> 625569 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[42 (1436499847447314432)]} 0 0
[junit4:junit4] 2> 625575 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[42 (1436499847450460160)]} 0 0
[junit4:junit4] 2> 625576 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[42 (1436499847450460160)]} 0 4
[junit4:junit4] 2> 625579 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[43 (1436499847457800192)]} 0 0
[junit4:junit4] 2> 625587 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[43 (1436499847463043072)]} 0 0
[junit4:junit4] 2> 625588 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[43 (1436499847463043072)]} 0 4
[junit4:junit4] 2> 625589 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[43]} 0 7
[junit4:junit4] 2> 625591 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[44 (1436499847470383104)]} 0 0
[junit4:junit4] 2> 625597 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[44 (1436499847473528832)]} 0 0
[junit4:junit4] 2> 625598 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[44 (1436499847473528832)]} 0 4
[junit4:junit4] 2> 625601 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[45 (1436499847480868864)]} 0 0
[junit4:junit4] 2> 625607 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[45 (1436499847484014592)]} 0 0
[junit4:junit4] 2> 625608 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[45 (1436499847484014592)]} 0 4
[junit4:junit4] 2> 625611 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[46 (1436499847491354624)]} 0 0
[junit4:junit4] 2> 625619 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[46 (1436499847496597504)]} 0 0
[junit4:junit4] 2> 625620 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[46 (1436499847496597504)]} 0 4
[junit4:junit4] 2> 625621 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[46]} 0 7
[junit4:junit4] 2> 625623 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[47 (1436499847503937536)]} 0 0
[junit4:junit4] 2> 625631 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[47 (1436499847509180416)]} 0 0
[junit4:junit4] 2> 625632 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[47 (1436499847509180416)]} 0 4
[junit4:junit4] 2> 625633 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[47]} 0 7
[junit4:junit4] 2> 625635 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[48 (1436499847516520448)]} 0 0
[junit4:junit4] 2> 625643 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[48 (1436499847521763328)]} 0 0
[junit4:junit4] 2> 625644 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[48 (1436499847521763328)]} 0 4
[junit4:junit4] 2> 625645 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[48]} 0 7
[junit4:junit4] 2> 625647 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[49 (1436499847529103360)]} 0 0
[junit4:junit4] 2> 625655 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[49 (1436499847534346240)]} 0 0
[junit4:junit4] 2> 625656 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[49 (1436499847534346240)]} 0 4
[junit4:junit4] 2> 625657 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[49]} 0 7
[junit4:junit4] 2> 625659 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[50 (1436499847541686272)]} 0 0
[junit4:junit4] 2> 625665 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[50 (1436499847544832000)]} 0 0
[junit4:junit4] 2> 625666 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[50 (1436499847544832000)]} 0 4
[junit4:junit4] 2> 625669 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[51 (1436499847552172032)]} 0 0
[junit4:junit4] 2> 625675 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[51 (1436499847555317760)]} 0 0
[junit4:junit4] 2> 625676 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[51 (1436499847555317760)]} 0 4
[junit4:junit4] 2> 625679 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[52 (1436499847562657792)]} 0 0
[junit4:junit4] 2> 625685 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[52 (1436499847565803520)]} 0 0
[junit4:junit4] 2> 625686 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[52 (1436499847565803520)]} 0 4
[junit4:junit4] 2> 625689 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[53 (1436499847573143552)]} 0 0
[junit4:junit4] 2> 625695 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[53 (1436499847576289280)]} 0 0
[junit4:junit4] 2> 625696 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[53 (1436499847576289280)]} 0 4
[junit4:junit4] 2> 625699 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[54 (1436499847583629312)]} 0 0
[junit4:junit4] 2> 625707 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[54 (1436499847588872192)]} 0 0
[junit4:junit4] 2> 625708 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[54 (1436499847588872192)]} 0 4
[junit4:junit4] 2> 625709 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[54]} 0 7
[junit4:junit4] 2> 625711 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[55 (1436499847596212224)]} 0 0
[junit4:junit4] 2> 625719 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[55 (1436499847601455104)]} 0 0
[junit4:junit4] 2> 625720 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[55 (1436499847601455104)]} 0 4
[junit4:junit4] 2> 625721 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[55]} 0 7
[junit4:junit4] 2> 625723 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[56 (1436499847608795136)]} 0 0
[junit4:junit4] 2> 625729 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[56 (1436499847611940864)]} 0 0
[junit4:junit4] 2> 625730 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[56 (1436499847611940864)]} 0 4
[junit4:junit4] 2> 625733 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[57 (1436499847619280896)]} 0 0
[junit4:junit4] 2> 625739 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[57 (1436499847622426624)]} 0 0
[junit4:junit4] 2> 625740 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[57 (1436499847622426624)]} 0 4
[junit4:junit4] 2> 625743 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[58 (1436499847629766656)]} 0 0
[junit4:junit4] 2> 625750 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[58 (1436499847632912384)]} 0 1
[junit4:junit4] 2> 625751 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[58 (1436499847632912384)]} 0 5
[junit4:junit4] 2> 625754 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[59 (1436499847640252416)]} 0 1
[junit4:junit4] 2> 625759 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[59 (1436499847643398144)]} 0 0
[junit4:junit4] 2> 625760 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[59 (1436499847643398144)]} 0 4
[junit4:junit4] 2> 625764 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[60 (1436499847650738176)]} 0 1
[junit4:junit4] 2> 625769 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[60 (1436499847653883904)]} 0 0
[junit4:junit4] 2> 625770 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[60 (1436499847653883904)]} 0 4
[junit4:junit4] 2> 625774 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[61 (1436499847661223936)]} 0 1
[junit4:junit4] 2> 625779 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[61 (1436499847664369664)]} 0 0
[junit4:junit4] 2> 625780 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[61 (1436499847664369664)]} 0 4
[junit4:junit4] 2> 625783 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[62 (1436499847671709696)]} 0 0
[junit4:junit4] 2> 625791 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[62 (1436499847676952576)]} 0 0
[junit4:junit4] 2> 625792 T1451 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[62 (1436499847676952576)]} 0 4
[junit4:junit4] 2> 625793 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[62]} 0 7
[junit4:junit4] 2> 625796 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[63 (1436499847685341184)]} 0 0
[junit4:junit4] 2> 625802 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[63 (1436499847688486912)]} 0 0
[junit4:junit4] 2> 625803 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[63 (1436499847688486912)]} 0 4
[junit4:junit4] 2> 625806 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[64 (1436499847695826944)]} 0 0
[junit4:junit4] 2> 625812 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[64 (1436499847698972672)]} 0 0
[junit4:junit4] 2> 625813 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[64 (1436499847698972672)]} 0 4
[junit4:junit4] 2> 625816 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[65 (1436499847706312704)]} 0 0
[junit4:junit4] 2> 625822 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[65 (1436499847709458432)]} 0 0
[junit4:junit4] 2> 625823 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[65 (1436499847709458432)]} 0 4
[junit4:junit4] 2> 625826 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[66 (1436499847716798464)]} 0 0
[junit4:junit4] 2> 625832 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[66 (1436499847719944192)]} 0 0
[junit4:junit4] 2> 625833 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[66 (1436499847719944192)]} 0 4
[junit4:junit4] 2> 625836 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[67 (1436499847727284224)]} 0 0
[junit4:junit4] 2> 625844 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[67 (1436499847732527104)]} 0 0
[junit4:junit4] 2> 625845 T1451 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[67 (1436499847732527104)]} 0 4
[junit4:junit4] 2> 625846 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[67]} 0 7
[junit4:junit4] 2> 625848 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[68 (1436499847739867136)]} 0 0
[junit4:junit4] 2> 625856 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[68 (1436499847745110016)]} 0 0
[junit4:junit4] 2> 625857 T1451 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[68 (1436499847745110016)]} 0 4
[junit4:junit4] 2> 625858 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[68]} 0 7
[junit4:junit4] 2> 625860 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[69 (1436499847752450048)]} 0 0
[junit4:junit4] 2> 625868 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[69 (1436499847757692928)]} 0 0
[junit4:junit4] 2> 625869 T1451 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[69 (1436499847757692928)]} 0 4
[junit4:junit4] 2> 625870 T1435 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[69]} 0 7
[junit4:junit4] 2> 625873 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[70 (1436499847766081536)]} 0 0
[junit4:junit4] 2> 625881 T1463 C428 P31637 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27681/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[70 (1436499847771324416)]} 0 0
[junit4:junit4] 2> 625882 T1437 C429 P27681 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[70 (1436499847771324416)]} 0 4
[junit4:junit4] 2> 625883 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[70]} 0 7
[junit4:junit4] 2> 625885 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[71 (1436499847778664448)]} 0 0
[junit4:junit4] 2> 625891 T1482 C427 P40188 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:27688/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[71 (1436499847781810176)]} 0 0
[junit4:junit4] 2> 625892 T1453 C430 P27688 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[71 (1436499847781810176)]} 0 4
[junit4:junit4] 2> 625895 T1411 C431 P27677 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[72 (143649984778
[...truncated too long message...]
.request(HttpSolrServer.java:180)
[junit4:junit4] > at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
[junit4:junit4] > at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:136)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 781679 T1399 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 808170 T1490 oasu.PeerSync.handleResponse WARN PeerSync: core=collection1 url=http://127.0.0.1:40188 couldn't connect to http://127.0.0.1:27688/collection1/, counting as success
[junit4:junit4] 2> 808171 T1490 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:40188 DONE. sync succeeded
[junit4:junit4] 2> 808171 T1490 oasc.SyncStrategy.syncReplicas We have been closed, won't attempt to sync replicas back to leader
[junit4:junit4] 2> 808172 T1490 oasc.ShardLeaderElectionContext.runLeaderProcess Sync was not a success but no one else is active! I am the leader
[junit4:junit4] 2> 808172 T1490 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40188/collection1/
[junit4:junit4] 2> 808173 T1490 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@3e20666e
[junit4:junit4] 2> 808174 T1490 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=4,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=704,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 808174 T1490 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 808175 T1490 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 808175 T1490 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 808178 T1490 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 808178 T1490 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 808179 T1490 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty4/index;done=false>>]
[junit4:junit4] 2> 808179 T1490 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty4/index
[junit4:junit4] 2> 808179 T1490 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty4;done=false>>]
[junit4:junit4] 2> 808180 T1490 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369952983002/jetty4
[junit4:junit4] 2> 808180 T1490 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 808180 T1490 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election/89781238694739980-127.0.0.1:40188__collection1-n_0000000001
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:152)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:149)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:149)
[junit4:junit4] 2> at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:63)
[junit4:junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:260)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:156)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:100)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> 808181 T1490 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 808181 T1490 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 808182 T1490 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 808182 T1490 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 808182 T1490 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 808182 T1490 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=nl_NL, timezone=Etc/GMT-5
[junit4:junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=3,free=157726600,total=480247808
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestBadConfig, LegacyHTMLStripCharFilterTest, DistributedSpellCheckComponentTest, TestPerFieldSimilarity, TestSearchPerf, BasicFunctionalityTest, RecoveryZkTest, TestReversedWildcardFilterFactory, SOLR749Test, TestSweetSpotSimilarityFactory, LukeRequestHandlerTest, TestFastWriter, QueryElevationComponentTest, TestWriterPerf, TestFieldTypeCollectionResource, TestSurroundQueryParser, OverseerCollectionProcessorTest, SuggesterFSTTest, TestFieldResource, TestLRUCache, FileBasedSpellCheckerTest, TestJoin, DirectSolrConnectionTest, UnloadDistributedZkTest, LoggingHandlerTest, DateFieldTest, TestSolrDeletionPolicy1, TestDocumentBuilder, CoreAdminHandlerTest, FieldMutatingUpdateProcessorTest, SuggesterTest, UUIDFieldTest, BasicDistributedZkTest, SimpleFacetsTest, TestExtendedDismaxParser, MinimalSchemaTest, ShowFileRequestHandlerTest, TestUniqueKeyFieldResource, TestQueryUtils, TestCoreContainer, TestCloudManagedSchemaAddField, ChaosMonkeyShardSplitTest]
[junit4:junit4] Completed on J0 in 210.22s, 1 test, 1 error <<< FAILURES!
[...truncated 604 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:380: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:360: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/common-build.xml:437: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1240: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:884: There were test failures: 295 suites, 1232 tests, 1 error, 13 ignored (7 assumptions)
Total time: 54 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-Tests-trunk-Java7 - Build # 4030 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-trunk-Java7/4030/
2 tests failed.
REGRESSION: org.apache.solr.cloud.ShardSplitTest.testDistribSearch
Error Message:
Server at http://127.0.0.1:38300 returned non ok status:500, message:Server Error
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:38300 returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([2FC1E187373C5811:AE276F9F4063382D]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
at org.apache.solr.cloud.ShardSplitTest.doTest(ShardSplitTest.java:133)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
REGRESSION: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:27106/ogmsi/tl/collection1lastClient and got 5 from http://127.0.0.1:28895/ogmsi/tl/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:27106/ogmsi/tl/collection1lastClient and got 5 from http://127.0.0.1:28895/ogmsi/tl/collection1
at __randomizedtesting.SeedInfo.seed([2FC1E187373C5811:AE276F9F4063382D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9460 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4] 2> 210272 T411 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ogmsi/tl
[junit4:junit4] 2> 210276 T411 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1369992701963
[junit4:junit4] 2> 210277 T411 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 210278 T412 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 210379 T411 oasc.ZkTestServer.run start zk server on port:51191
[junit4:junit4] 2> 210380 T411 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 210384 T418 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6be98e3e name:ZooKeeperConnection Watcher:127.0.0.1:51191 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 210384 T411 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 210385 T411 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 210407 T411 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 210410 T420 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74ee46b3 name:ZooKeeperConnection Watcher:127.0.0.1:51191/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 210410 T411 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 210410 T411 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 210422 T411 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 210426 T411 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 210430 T411 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 210434 T411 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 210434 T411 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 210446 T411 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 210447 T411 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 210555 T411 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 210556 T411 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 210560 T411 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 210561 T411 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 210565 T411 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 210566 T411 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 210570 T411 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 210571 T411 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 210575 T411 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 210576 T411 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 210580 T411 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 210581 T411 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 210585 T411 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 210586 T411 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 210859 T411 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 210863 T411 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:36610
[junit4:junit4] 2> 210864 T411 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 210865 T411 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 210865 T411 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1369992702277
[junit4:junit4] 2> 210866 T411 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1369992702277/solr.xml
[junit4:junit4] 2> 210866 T411 oasc.CoreContainer.<init> New CoreContainer 1814440606
[junit4:junit4] 2> 210867 T411 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1369992702277/'
[junit4:junit4] 2> 210868 T411 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1369992702277/'
[junit4:junit4] 2> 210996 T411 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 210997 T411 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 210997 T411 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 210998 T411 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 210998 T411 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 210999 T411 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 210999 T411 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 211000 T411 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 211000 T411 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 211001 T411 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 211020 T411 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 211021 T411 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51191/solr
[junit4:junit4] 2> 211021 T411 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 211023 T411 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 211026 T431 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4bb6a05d name:ZooKeeperConnection Watcher:127.0.0.1:51191 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 211027 T411 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 211040 T411 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 211050 T411 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 211052 T433 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@da65f1d name:ZooKeeperConnection Watcher:127.0.0.1:51191/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 211052 T411 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 211060 T411 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 211066 T411 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 211071 T411 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 211074 T411 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36610_ogmsi%2Ftl
[junit4:junit4] 2> 211081 T411 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36610_ogmsi%2Ftl
[junit4:junit4] 2> 211086 T411 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 211102 T411 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 211105 T411 oasc.Overseer.start Overseer (id=89783841716371459-127.0.0.1:36610_ogmsi%2Ftl-n_0000000000) starting
[junit4:junit4] 2> 211118 T411 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 211129 T435 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 211130 T411 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 211133 T411 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 211136 T411 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 211140 T434 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 211150 T436 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1369992702277/collection1
[junit4:junit4] 2> 211150 T436 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 211151 T436 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 211151 T436 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 211154 T436 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1369992702277/collection1/'
[junit4:junit4] 2> 211155 T436 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1369992702277/collection1/lib/README' to classloader
[junit4:junit4] 2> 211156 T436 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1369992702277/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 211209 T436 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 211270 T436 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 211373 T436 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 211380 T436 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 212007 T436 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 212022 T436 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 212026 T436 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 212050 T436 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 212056 T436 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 212061 T436 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 212062 T436 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 212063 T436 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 212063 T436 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 212065 T436 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 212065 T436 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 212065 T436 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 212113 T436 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1369992702277/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1369992701963/control/data/
[junit4:junit4] 2> 212113 T436 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@55877de5
[junit4:junit4] 2> 212113 T436 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 212115 T436 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/control/data
[junit4:junit4] 2> 212116 T436 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1369992701963/control/data/index/
[junit4:junit4] 2> 212117 T436 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1369992701963/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 212118 T436 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/control/data/index
[junit4:junit4] 2> 212126 T436 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e3eb0d8; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 212127 T436 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 212130 T436 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 212130 T436 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 212131 T436 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 212132 T436 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 212132 T436 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 212132 T436 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 212133 T436 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 212133 T436 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 212134 T436 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 212150 T436 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 212159 T436 oass.SolrIndexSearcher.<init> Opening Searcher@249905e4 main
[junit4:junit4] 2> 212161 T436 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 212161 T436 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 212165 T437 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@249905e4 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 212167 T436 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 212167 T436 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 212645 T434 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 212646 T434 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:36610_ogmsi%2Ftl",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:36610/ogmsi/tl"}
[junit4:junit4] 2> 212647 T434 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 212680 T434 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 212706 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 213171 T436 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 213171 T436 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36610/ogmsi/tl collection:control_collection shard:shard1
[junit4:junit4] 2> 213184 T436 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 213204 T436 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 213228 T436 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 213232 T436 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 213233 T436 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 213239 T436 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:36610/ogmsi/tl/collection1/
[junit4:junit4] 2> 213240 T436 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 213240 T436 oasc.SyncStrategy.syncToMe http://127.0.0.1:36610/ogmsi/tl/collection1/ has no replicas
[junit4:junit4] 2> 213240 T436 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:36610/ogmsi/tl/collection1/
[junit4:junit4] 2> 213241 T436 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 214212 T434 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 214262 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 214283 T436 oasc.ZkController.register We are http://127.0.0.1:36610/ogmsi/tl/collection1/ and leader is http://127.0.0.1:36610/ogmsi/tl/collection1/
[junit4:junit4] 2> 214283 T436 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36610/ogmsi/tl
[junit4:junit4] 2> 214283 T436 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 214284 T436 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 214284 T436 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 214286 T436 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 214288 T411 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 214288 T411 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 214289 T411 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 214304 T411 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 214315 T411 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 214319 T440 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14a1b15 name:ZooKeeperConnection Watcher:127.0.0.1:51191/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 214319 T411 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 214322 T411 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 214328 T411 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 214625 T411 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 214628 T411 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:30366
[junit4:junit4] 2> 214629 T411 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 214630 T411 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 214630 T411 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1369992706015
[junit4:junit4] 2> 214631 T411 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1369992706015/solr.xml
[junit4:junit4] 2> 214631 T411 oasc.CoreContainer.<init> New CoreContainer 1018480479
[junit4:junit4] 2> 214632 T411 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1369992706015/'
[junit4:junit4] 2> 214632 T411 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1369992706015/'
[junit4:junit4] 2> 214760 T411 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 214761 T411 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 214761 T411 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 214762 T411 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 214762 T411 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 214763 T411 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 214763 T411 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 214764 T411 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 214764 T411 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 214765 T411 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 214784 T411 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 214784 T411 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51191/solr
[junit4:junit4] 2> 214785 T411 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 214786 T411 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 214818 T451 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a2c2f09 name:ZooKeeperConnection Watcher:127.0.0.1:51191 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 214818 T411 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 214822 T411 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 214836 T411 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 214839 T453 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10147024 name:ZooKeeperConnection Watcher:127.0.0.1:51191/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 214839 T411 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 214851 T411 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 215776 T434 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 215777 T434 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:36610_ogmsi%2Ftl",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:36610/ogmsi/tl"}
[junit4:junit4] 2> 215793 T453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 215793 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 215793 T440 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 215856 T411 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:30366_ogmsi%2Ftl
[junit4:junit4] 2> 215858 T411 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:30366_ogmsi%2Ftl
[junit4:junit4] 2> 215862 T440 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 215862 T433 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 215862 T440 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 215862 T453 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 215863 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 215863 T453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 215882 T454 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1369992706015/collection1
[junit4:junit4] 2> 215882 T454 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 215883 T454 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 215883 T454 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 215885 T454 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1369992706015/collection1/'
[junit4:junit4] 2> 215887 T454 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1369992706015/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 215887 T454 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1369992706015/collection1/lib/README' to classloader
[junit4:junit4] 2> 215940 T454 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 216012 T454 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 216115 T454 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 216122 T454 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 216761 T454 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 216788 T454 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 216793 T454 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 216817 T454 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 216822 T454 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 216828 T454 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 216829 T454 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 216830 T454 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 216830 T454 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 216831 T454 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 216832 T454 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 216832 T454 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 216832 T454 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1369992706015/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/
[junit4:junit4] 2> 216833 T454 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@55877de5
[junit4:junit4] 2> 216833 T454 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 216835 T454 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1
[junit4:junit4] 2> 216836 T454 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index/
[junit4:junit4] 2> 216837 T454 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 216837 T454 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index
[junit4:junit4] 2> 216845 T454 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 216846 T454 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 216849 T454 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 216850 T454 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 216850 T454 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 216851 T454 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 216852 T454 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 216852 T454 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 216852 T454 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 216853 T454 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 216854 T454 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 216872 T454 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 216882 T454 oass.SolrIndexSearcher.<init> Opening Searcher@138bb487 main
[junit4:junit4] 2> 216884 T454 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 216884 T454 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 216888 T455 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@138bb487 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 216891 T454 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 216891 T454 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 217301 T434 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 217302 T434 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:30366_ogmsi%2Ftl",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30366/ogmsi/tl"}
[junit4:junit4] 2> 217302 T434 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 217302 T434 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 217315 T453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 217315 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 217315 T440 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 217894 T454 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 217894 T454 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:30366/ogmsi/tl collection:collection1 shard:shard1
[junit4:junit4] 2> 217895 T454 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 217914 T454 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 217925 T454 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 217926 T454 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 217926 T454 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:30366/ogmsi/tl/collection1/
[junit4:junit4] 2> 217926 T454 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 217927 T454 oasc.SyncStrategy.syncToMe http://127.0.0.1:30366/ogmsi/tl/collection1/ has no replicas
[junit4:junit4] 2> 217927 T454 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:30366/ogmsi/tl/collection1/
[junit4:junit4] 2> 217927 T454 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 218822 T434 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 218841 T453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 218841 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 218841 T440 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 218856 T454 oasc.ZkController.register We are http://127.0.0.1:30366/ogmsi/tl/collection1/ and leader is http://127.0.0.1:30366/ogmsi/tl/collection1/
[junit4:junit4] 2> 218856 T454 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:30366/ogmsi/tl
[junit4:junit4] 2> 218856 T454 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 218856 T454 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 218857 T454 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 218860 T454 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 218861 T411 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 218862 T411 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 218862 T411 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 219164 T411 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 219168 T411 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:12293
[junit4:junit4] 2> 219168 T411 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 219169 T411 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 219169 T411 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1369992710566
[junit4:junit4] 2> 219170 T411 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1369992710566/solr.xml
[junit4:junit4] 2> 219170 T411 oasc.CoreContainer.<init> New CoreContainer 1275831735
[junit4:junit4] 2> 219171 T411 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1369992710566/'
[junit4:junit4] 2> 219172 T411 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1369992710566/'
[junit4:junit4] 2> 219304 T411 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 219305 T411 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 219306 T411 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 219306 T411 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 219307 T411 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 219307 T411 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 219308 T411 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 219308 T411 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 219309 T411 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 219309 T411 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 219326 T411 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 219327 T411 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51191/solr
[junit4:junit4] 2> 219328 T411 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 219329 T411 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 219332 T467 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@301e026a name:ZooKeeperConnection Watcher:127.0.0.1:51191 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 219332 T411 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 219335 T411 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 219347 T411 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 219350 T469 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11364b81 name:ZooKeeperConnection Watcher:127.0.0.1:51191/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 219350 T411 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 219362 T411 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 220356 T434 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 220358 T434 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:30366_ogmsi%2Ftl",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:30366/ogmsi/tl"}
[junit4:junit4] 2> 220385 T411 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:12293_ogmsi%2Ftl
[junit4:junit4] 2> 220387 T411 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:12293_ogmsi%2Ftl
[junit4:junit4] 2> 220401 T453 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 220401 T469 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 220401 T453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 220401 T440 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 220401 T469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 220402 T440 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 220408 T453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 220409 T440 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 220410 T433 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 220410 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 220412 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 220415 T470 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1369992710566/collection1
[junit4:junit4] 2> 220415 T470 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 220416 T470 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 220416 T470 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 220418 T470 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1369992710566/collection1/'
[junit4:junit4] 2> 220420 T470 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1369992710566/collection1/lib/README' to classloader
[junit4:junit4] 2> 220420 T470 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1369992710566/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 220472 T470 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 220533 T470 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 220635 T470 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 220642 T470 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 221277 T470 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 221292 T470 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 221296 T470 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 221320 T470 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 221325 T470 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 221331 T470 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 221332 T470 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 221333 T470 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 221333 T470 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 221335 T470 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 221335 T470 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 221335 T470 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 221336 T470 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1369992710566/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2/
[junit4:junit4] 2> 221336 T470 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@55877de5
[junit4:junit4] 2> 221337 T470 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 221338 T470 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2
[junit4:junit4] 2> 221339 T470 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2/index/
[junit4:junit4] 2> 221340 T470 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 221341 T470 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2/index
[junit4:junit4] 2> 221358 T470 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d854b15; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 221359 T470 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 221362 T470 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 221363 T470 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 221364 T470 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 221364 T470 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 221365 T470 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 221365 T470 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 221366 T470 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 221366 T470 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 221367 T470 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 221386 T470 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 221396 T470 oass.SolrIndexSearcher.<init> Opening Searcher@439a59e5 main
[junit4:junit4] 2> 221398 T470 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 221398 T470 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 221403 T471 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@439a59e5 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 221406 T470 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 221406 T470 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 221913 T434 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 221915 T434 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:12293_ogmsi%2Ftl",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:12293/ogmsi/tl"}
[junit4:junit4] 2> 221915 T434 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 221915 T434 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 221928 T469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 221929 T453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 221929 T440 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 221929 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 222413 T470 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 222413 T470 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:12293/ogmsi/tl collection:collection1 shard:shard1
[junit4:junit4] 2> 222417 T470 oasc.ZkController.register We are http://127.0.0.1:12293/ogmsi/tl/collection1/ and leader is http://127.0.0.1:30366/ogmsi/tl/collection1/
[junit4:junit4] 2> 222418 T470 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:12293/ogmsi/tl
[junit4:junit4] 2> 222418 T470 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 222418 T470 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C268 name=collection1 org.apache.solr.core.SolrCore@62df7f31 url=http://127.0.0.1:12293/ogmsi/tl/collection1 node=127.0.0.1:12293_ogmsi%2Ftl C268_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:12293_ogmsi%2Ftl, base_url=http://127.0.0.1:12293/ogmsi/tl}
[junit4:junit4] 2> 222426 T472 C268 P12293 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 222427 T470 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 222427 T472 C268 P12293 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 222428 T472 C268 P12293 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 222428 T472 C268 P12293 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 222429 T411 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 222429 T411 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 222430 T411 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 222430 T472 C268 P12293 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 222471 T443 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 222737 T411 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 222740 T411 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:27106
[junit4:junit4] 2> 222742 T411 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 222742 T411 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 222743 T411 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1369992714137
[junit4:junit4] 2> 222743 T411 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1369992714137/solr.xml
[junit4:junit4] 2> 222744 T411 oasc.CoreContainer.<init> New CoreContainer 1252991878
[junit4:junit4] 2> 222744 T411 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1369992714137/'
[junit4:junit4] 2> 222745 T411 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1369992714137/'
[junit4:junit4] 2> 222882 T411 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 222883 T411 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 222883 T411 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 222884 T411 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 222884 T411 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 222885 T411 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 222885 T411 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 222886 T411 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 222886 T411 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 222887 T411 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 222907 T411 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 222908 T411 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51191/solr
[junit4:junit4] 2> 222909 T411 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 222910 T411 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 222913 T484 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50b93e10 name:ZooKeeperConnection Watcher:127.0.0.1:51191 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 222914 T411 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 222924 T411 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 222943 T411 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 222946 T486 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d6e44ca name:ZooKeeperConnection Watcher:127.0.0.1:51191/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 222946 T411 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 222953 T411 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 223435 T434 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 223437 T434 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:12293_ogmsi%2Ftl",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:12293/ogmsi/tl"}
[junit4:junit4] 2> 223451 T453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 223451 T469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 223451 T486 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 223451 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 223451 T440 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 223474 T443 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 223474 T443 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=3&state=recovering&nodeName=127.0.0.1:12293_ogmsi%252Ftl&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1012
[junit4:junit4] 2> 223959 T411 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:27106_ogmsi%2Ftl
[junit4:junit4] 2> 223967 T411 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:27106_ogmsi%2Ftl
[junit4:junit4] 2> 223972 T440 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 223972 T453 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 223973 T486 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 223973 T433 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 223972 T440 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 223972 T469 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 223973 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 223973 T486 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 223973 T453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 223974 T469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 223995 T487 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1369992714137/collection1
[junit4:junit4] 2> 223996 T487 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 223996 T487 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 223997 T487 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 223999 T487 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1369992714137/collection1/'
[junit4:junit4] 2> 224000 T487 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1369992714137/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 224001 T487 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1369992714137/collection1/lib/README' to classloader
[junit4:junit4] 2> 224053 T487 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 224113 T487 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 224216 T487 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 224223 T487 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 224847 T487 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 224861 T487 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 224865 T487 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 224889 T487 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 224894 T487 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 224899 T487 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 224901 T487 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 224902 T487 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 224902 T487 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 224903 T487 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 224904 T487 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 224904 T487 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 224904 T487 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1369992714137/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3/
[junit4:junit4] 2> 224905 T487 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@55877de5
[junit4:junit4] 2> 224905 T487 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 224906 T487 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3
[junit4:junit4] 2> 224907 T487 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3/index/
[junit4:junit4] 2> 224908 T487 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 224909 T487 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3/index
[junit4:junit4] 2> 224916 T487 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5afde945; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 224916 T487 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 224920 T487 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 224920 T487 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 224921 T487 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 224922 T487 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 224922 T487 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 224923 T487 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 224923 T487 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 224924 T487 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 224924 T487 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 224940 T487 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 224950 T487 oass.SolrIndexSearcher.<init> Opening Searcher@2fa9dd0e main
[junit4:junit4] 2> 224951 T487 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 224952 T487 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 224956 T488 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2fa9dd0e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 224959 T487 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 224959 T487 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> ASYNC NEW_CORE C269 name=collection1 org.apache.solr.core.SolrCore@62df7f31 url=http://127.0.0.1:12293/ogmsi/tl/collection1 node=127.0.0.1:12293_ogmsi%2Ftl C269_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:12293_ogmsi%2Ftl, base_url=http://127.0.0.1:12293/ogmsi/tl}
[junit4:junit4] 2> 225476 T472 C269 P12293 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:30366/ogmsi/tl/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 225488 T472 C269 P12293 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 225515 T472 C269 P12293 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:12293/ogmsi/tl START replicas=[http://127.0.0.1:30366/ogmsi/tl/collection1/] nUpdates=100
[junit4:junit4] 2> 225516 T472 C269 P12293 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 225516 T472 C269 P12293 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 225517 T472 C269 P12293 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 225517 T472 C269 P12293 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 225517 T472 C269 P12293 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 225517 T472 C269 P12293 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:30366/ogmsi/tl/collection1/. core=collection1
[junit4:junit4] 2> 225518 T472 C269 P12293 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C270 name=collection1 org.apache.solr.core.SolrCore@1e6a3d63 url=http://127.0.0.1:30366/ogmsi/tl/collection1 node=127.0.0.1:30366_ogmsi%2Ftl C270_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:30366_ogmsi%2Ftl, base_url=http://127.0.0.1:30366/ogmsi/tl, leader=true}
[junit4:junit4] 2> 225526 T444 C270 P30366 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 225553 T445 C270 P30366 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 225558 T445 C270 P30366 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 225559 T445 C270 P30366 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 225563 T445 C270 P30366 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 225564 T445 C270 P30366 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 225566 T445 C270 P30366 oass.SolrIndexSearcher.<init> Opening Searcher@5dcf4a6 realtime
[junit4:junit4] 2> 225566 T445 C270 P30366 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 225567 T445 C270 P30366 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 36
[junit4:junit4] 2> 225568 T472 C269 P12293 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 225568 T472 C269 P12293 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4] 2> 225599 T472 C269 P12293 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 225602 T446 C270 P30366 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 225603 T446 C270 P30366 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 225604 T472 C269 P12293 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 225604 T472 C269 P12293 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 225604 T472 C269 P12293 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 225609 T446 C270 P30366 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=2
[junit4:junit4] 2> 225610 T472 C269 P12293 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 225612 T472 C269 P12293 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2/index.20130531043157296
[junit4:junit4] 2> 225613 T472 C269 P12293 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2/index.20130531043157296 lockFactory=org.apache.lucene.store.NativeFSLockFactory@352b747c; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 225630 T446 C270 P30366 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=6
[junit4:junit4] 2> 225648 T472 C269 P12293 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 225652 T472 C269 P12293 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 225652 T472 C269 P12293 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 225664 T472 C269 P12293 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d854b15; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d854b15; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 225665 T472 C269 P12293 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 225666 T472 C269 P12293 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 225667 T472 C269 P12293 oass.SolrIndexSearcher.<init> Opening Searcher@58ef86f0 main
[junit4:junit4] 2> 225668 T471 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@58ef86f0 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 225669 T472 C269 P12293 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2/index.20130531043157296 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2/index.20130531043157296;done=true>>]
[junit4:junit4] 2> 225669 T472 C269 P12293 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2/index.20130531043157296
[junit4:junit4] 2> 225670 T472 C269 P12293 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2/index.20130531043157296
[junit4:junit4] 2> 225680 T472 C269 P12293 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 225680 T472 C269 P12293 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 225680 T472 C269 P12293 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 225681 T472 C269 P12293 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 225683 T472 C269 P12293 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 226460 T434 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 226461 T434 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:27106_ogmsi%2Ftl",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:27106/ogmsi/tl"}
[junit4:junit4] 2> 226462 T434 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 226462 T434 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 226492 T434 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:12293_ogmsi%2Ftl",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:12293/ogmsi/tl"}
[junit4:junit4] 2> 226502 T453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 226502 T486 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 226502 T469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 226502 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 226502 T440 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 226964 T487 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 226964 T487 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:27106/ogmsi/tl collection:collection1 shard:shard1
[junit4:junit4] 2> 226969 T487 oasc.ZkController.register We are http://127.0.0.1:27106/ogmsi/tl/collection1/ and leader is http://127.0.0.1:30366/ogmsi/tl/collection1/
[junit4:junit4] 2> 226970 T487 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:27106/ogmsi/tl
[junit4:junit4] 2> 226970 T487 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 226970 T487 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C271 name=collection1 org.apache.solr.core.SolrCore@4f8a312c url=http://127.0.0.1:27106/ogmsi/tl/collection1 node=127.0.0.1:27106_ogmsi%2Ftl C271_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:27106_ogmsi%2Ftl, base_url=http://127.0.0.1:27106/ogmsi/tl}
[junit4:junit4] 2> 226971 T491 C271 P27106 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 226972 T487 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 226972 T491 C271 P27106 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 226973 T491 C271 P27106 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 226973 T491 C271 P27106 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 226974 T411 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 226974 T411 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 226975 T491 C271 P27106 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 226975 T411 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 227003 T447 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 227270 T411 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 227273 T411 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:28895
[junit4:junit4] 2> 227274 T411 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 227275 T411 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 227275 T411 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1369992718686
[junit4:junit4] 2> 227276 T411 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1369992718686/solr.xml
[junit4:junit4] 2> 227276 T411 oasc.CoreContainer.<init> New CoreContainer 328532789
[junit4:junit4] 2> 227277 T411 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1369992718686/'
[junit4:junit4] 2> 227278 T411 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1369992718686/'
[junit4:junit4] 2> 227405 T411 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 227406 T411 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 227407 T411 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 227407 T411 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 227408 T411 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 227408 T411 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 227409 T411 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 227409 T411 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 227410 T411 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 227410 T411 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 227428 T411 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 227428 T411 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51191/solr
[junit4:junit4] 2> 227429 T411 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 227430 T411 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 227434 T503 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4cd72139 name:ZooKeeperConnection Watcher:127.0.0.1:51191 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 227434 T411 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 227446 T411 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 227459 T411 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 227461 T505 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@362642ed name:ZooKeeperConnection Watcher:127.0.0.1:51191/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 227461 T411 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 227472 T411 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 228018 T434 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 228020 T434 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:27106_ogmsi%2Ftl",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:27106/ogmsi/tl"}
[junit4:junit4] 2> 228026 T453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 228026 T486 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 228026 T469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 228026 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 228026 T440 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 228026 T505 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 228478 T411 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:28895_ogmsi%2Ftl
[junit4:junit4] 2> 228480 T411 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:28895_ogmsi%2Ftl
[junit4:junit4] 2> 228484 T505 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 228484 T453 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 228484 T433 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 228485 T486 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 228485 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 228485 T469 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 228484 T453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 228484 T440 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 228485 T469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 228485 T486 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 228486 T440 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 228487 T505 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 228495 T506 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1369992718686/collection1
[junit4:junit4] 2> 228495 T506 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 228496 T506 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 228496 T506 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 228498 T506 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1369992718686/collection1/'
[junit4:junit4] 2> 228500 T506 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1369992718686/collection1/lib/README' to classloader
[junit4:junit4] 2> 228500 T506 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1369992718686/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 228554 T506 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 228633 T506 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 228735 T506 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 228743 T506 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 229259 T447 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 229259 T447 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=4&state=recovering&nodeName=127.0.0.1:27106_ogmsi%252Ftl&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2257
[junit4:junit4] 2> 229287 T506 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 229298 T506 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 229300 T506 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 229320 T506 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 229324 T506 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 229328 T506 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 229330 T506 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 229330 T506 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 229330 T506 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 229331 T506 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 229332 T506 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 229332 T506 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 229332 T506 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1369992718686/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4/
[junit4:junit4] 2> 229333 T506 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@55877de5
[junit4:junit4] 2> 229333 T506 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 229334 T506 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4
[junit4:junit4] 2> 229334 T506 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4/index/
[junit4:junit4] 2> 229335 T506 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 229336 T506 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4/index
[junit4:junit4] 2> 229341 T506 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6164799f; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 229342 T506 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 229356 T506 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 229356 T506 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 229357 T506 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 229357 T506 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 229358 T506 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 229358 T506 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 229358 T506 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 229359 T506 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 229359 T506 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 229372 T506 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 229381 T506 oass.SolrIndexSearcher.<init> Opening Searcher@353ad96 main
[junit4:junit4] 2> 229382 T506 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 229382 T506 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 229386 T507 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@353ad96 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 229388 T506 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 229388 T506 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 229534 T434 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 229535 T434 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:28895_ogmsi%2Ftl",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:28895/ogmsi/tl"}
[junit4:junit4] 2> 229535 T434 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 229536 T434 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 229547 T453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 229547 T469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 229547 T505 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 229547 T486 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 229547 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 229547 T440 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 230391 T506 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 230391 T506 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:28895/ogmsi/tl collection:collection1 shard:shard1
[junit4:junit4] 2> 230424 T506 oasc.ZkController.register We are http://127.0.0.1:28895/ogmsi/tl/collection1/ and leader is http://127.0.0.1:30366/ogmsi/tl/collection1/
[junit4:junit4] 2> 230424 T506 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:28895/ogmsi/tl
[junit4:junit4] 2> 230424 T506 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 230425 T506 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C272 name=collection1 org.apache.solr.core.SolrCore@31c12b61 url=http://127.0.0.1:28895/ogmsi/tl/collection1 node=127.0.0.1:28895_ogmsi%2Ftl C272_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:28895_ogmsi%2Ftl, base_url=http://127.0.0.1:28895/ogmsi/tl}
[junit4:junit4] 2> 230425 T508 C272 P28895 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 230426 T508 C272 P28895 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 230426 T506 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 230426 T508 C272 P28895 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 230427 T508 C272 P28895 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 230428 T411 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0
[junit4:junit4] 2> 230428 T411 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 230429 T411 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 230433 T508 C272 P28895 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 230454 T411 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 230455 T448 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 230456 T411 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 230457 T411 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 230458 T411 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 231053 T434 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 231054 T434 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:28895_ogmsi%2Ftl",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:28895/ogmsi/tl"}
[junit4:junit4] 2> 231102 T453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 231102 T486 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 231102 T440 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 231102 T469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 231102 T505 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 231102 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> ASYNC NEW_CORE C273 name=collection1 org.apache.solr.core.SolrCore@4f8a312c url=http://127.0.0.1:27106/ogmsi/tl/collection1 node=127.0.0.1:27106_ogmsi%2Ftl C273_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:27106_ogmsi%2Ftl, base_url=http://127.0.0.1:27106/ogmsi/tl}
[junit4:junit4] 2> 231261 T491 C273 P27106 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:30366/ogmsi/tl/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 231261 T491 C273 P27106 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:27106/ogmsi/tl START replicas=[http://127.0.0.1:30366/ogmsi/tl/collection1/] nUpdates=100
[junit4:junit4] 2> 231262 T491 C273 P27106 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 231262 T491 C273 P27106 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 231262 T491 C273 P27106 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 231263 T491 C273 P27106 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 231263 T491 C273 P27106 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 231263 T491 C273 P27106 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:30366/ogmsi/tl/collection1/. core=collection1
[junit4:junit4] 2> 231263 T491 C273 P27106 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C274 name=collection1 org.apache.solr.core.SolrCore@1e6a3d63 url=http://127.0.0.1:30366/ogmsi/tl/collection1 node=127.0.0.1:30366_ogmsi%2Ftl C274_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:30366_ogmsi%2Ftl, base_url=http://127.0.0.1:30366/ogmsi/tl, leader=true}
[junit4:junit4] 2> 231265 T444 C274 P30366 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 231274 T449 C274 P30366 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 231277 T449 C274 P30366 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 231278 T449 C274 P30366 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 231279 T449 C274 P30366 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 231279 T449 C274 P30366 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 231280 T491 C273 P27106 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 231280 T491 C273 P27106 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 231283 T446 C274 P30366 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 231284 T491 C273 P27106 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4] 2> 231285 T491 C273 P27106 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 231285 T491 C273 P27106 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 231288 T446 C274 P30366 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=3&version=2} status=0 QTime=1
[junit4:junit4] 2> 231289 T491 C273 P27106 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 231290 T491 C273 P27106 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3/index.20130531043202975
[junit4:junit4] 2> 231291 T491 C273 P27106 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3/index.20130531043202975 lockFactory=org.apache.lucene.store.NativeFSLockFactory@71c05518; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 231294 T446 C274 P30366 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/replication params={file=segments_3&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=3} status=0 QTime=0
[junit4:junit4] 2> 231296 T491 C273 P27106 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 231298 T491 C273 P27106 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 231298 T491 C273 P27106 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 231302 T491 C273 P27106 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5afde945; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5afde945; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 231302 T491 C273 P27106 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 231302 T491 C273 P27106 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 231303 T491 C273 P27106 oass.SolrIndexSearcher.<init> Opening Searcher@26b1ba1a main
[junit4:junit4] 2> 231304 T488 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@26b1ba1a main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 231304 T491 C273 P27106 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3/index.20130531043202975 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3/index.20130531043202975;done=true>>]
[junit4:junit4] 2> 231305 T491 C273 P27106 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3/index.20130531043202975
[junit4:junit4] 2> 231305 T491 C273 P27106 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3/index.20130531043202975
[junit4:junit4] 2> 231306 T491 C273 P27106 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 231306 T491 C273 P27106 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 231306 T491 C273 P27106 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 231306 T491 C273 P27106 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 231308 T491 C273 P27106 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 231457 T448 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 231457 T448 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:28895_ogmsi%252Ftl&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4] 2> 231460 T411 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 232463 T411 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 232608 T434 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 232608 T434 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:27106_ogmsi%2Ftl",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:27106/ogmsi/tl"}
[junit4:junit4] 2> 232629 T440 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 232629 T469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 232629 T505 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 232629 T486 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 232629 T453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 232629 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> C272_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:28895_ogmsi%2Ftl, base_url=http://127.0.0.1:28895/ogmsi/tl}
[junit4:junit4] 2> 233459 T508 C272 P28895 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:30366/ogmsi/tl/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 233459 T508 C272 P28895 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:28895/ogmsi/tl START replicas=[http://127.0.0.1:30366/ogmsi/tl/collection1/] nUpdates=100
[junit4:junit4] 2> 233460 T508 C272 P28895 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 233460 T508 C272 P28895 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 233460 T508 C272 P28895 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 233460 T508 C272 P28895 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 233461 T508 C272 P28895 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 233461 T508 C272 P28895 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:30366/ogmsi/tl/collection1/. core=collection1
[junit4:junit4] 2> 233461 T508 C272 P28895 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 233463 T444 C274 P30366 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 233465 T411 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 233474 T443 C274 P30366 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 233479 T443 C274 P30366 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 233480 T443 C274 P30366 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 233481 T443 C274 P30366 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 233482 T443 C274 P30366 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 233483 T508 C272 P28895 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 233483 T508 C272 P28895 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 233485 T446 C274 P30366 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 233486 T508 C272 P28895 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4] 2> 233486 T508 C272 P28895 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 233486 T508 C272 P28895 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 233490 T446 C274 P30366 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=4&version=2} status=0 QTime=1
[junit4:junit4] 2> 233491 T508 C272 P28895 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 233493 T508 C272 P28895 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4/index.20130531043205178
[junit4:junit4] 2> 233494 T508 C272 P28895 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4/index.20130531043205178 lockFactory=org.apache.lucene.store.NativeFSLockFactory@432b55e; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 233497 T446 C274 P30366 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/replication params={file=segments_4&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=4} status=0 QTime=0
[junit4:junit4] 2> 233499 T508 C272 P28895 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 233501 T508 C272 P28895 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 233501 T508 C272 P28895 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 233505 T508 C272 P28895 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6164799f; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6164799f; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 233506 T508 C272 P28895 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 233506 T508 C272 P28895 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 233507 T508 C272 P28895 oass.SolrIndexSearcher.<init> Opening Searcher@12a0dd84 main
[junit4:junit4] 2> 233508 T507 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@12a0dd84 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 233508 T508 C272 P28895 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4/index.20130531043205178 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4/index.20130531043205178;done=true>>]
[junit4:junit4] 2> 233509 T508 C272 P28895 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4/index.20130531043205178
[junit4:junit4] 2> 233509 T508 C272 P28895 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4/index.20130531043205178
[junit4:junit4] 2> 233510 T508 C272 P28895 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 233510 T508 C272 P28895 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 233510 T508 C272 P28895 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 233510 T508 C272 P28895 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 233512 T508 C272 P28895 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 234136 T434 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 234137 T434 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:28895_ogmsi%2Ftl",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:28895/ogmsi/tl"}
[junit4:junit4] 2> 234204 T453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 234204 T486 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 234204 T469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 234204 T505 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 234204 T433 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 234204 T440 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 234469 T411 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 234470 T411 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C275 name=collection1 org.apache.solr.core.SolrCore@67f821aa url=http://127.0.0.1:36610/ogmsi/tl/collection1 node=127.0.0.1:36610_ogmsi%2Ftl C275_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:36610_ogmsi%2Ftl, base_url=http://127.0.0.1:36610/ogmsi/tl, leader=true}
[junit4:junit4] 2> 234485 T423 C275 P36610 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 234491 T423 C275 P36610 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e3eb0d8; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 234492 T423 C275 P36610 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 234502 T423 C275 P36610 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e3eb0d8; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e3eb0d8; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 234502 T423 C275 P36610 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 234505 T423 C275 P36610 oass.SolrIndexSearcher.<init> Opening Searcher@6786c507 main
[junit4:junit4] 2> 234505 T423 C275 P36610 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 234506 T437 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6786c507 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 234507 T423 C275 P36610 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 22
[junit4:junit4] 2> 234522 T445 C274 P30366 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 234527 T445 C274 P30366 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=4
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 234528 T445 C274 P30366 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 234530 T445 C274 P30366 oass.SolrIndexSearcher.<init> Opening Searcher@1de6e9d6 main
[junit4:junit4] 2> 234530 T445 C274 P30366 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 234531 T455 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1de6e9d6 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 234552 T445 C274 P30366 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:12293/ogmsi/tl/collection1/, StdNode: http://127.0.0.1:27106/ogmsi/tl/collection1/, StdNode: http://127.0.0.1:28895/ogmsi/tl/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> C273_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:27106_ogmsi%2Ftl, base_url=http://127.0.0.1:27106/ogmsi/tl}
[junit4:junit4] 2> 234587 T476 C273 P27106 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C276 name=collection1 org.apache.solr.core.SolrCore@62df7f31 url=http://127.0.0.1:12293/ogmsi/tl/collection1 node=127.0.0.1:12293_ogmsi%2Ftl C276_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:12293_ogmsi%2Ftl, base_url=http://127.0.0.1:12293/ogmsi/tl}
[junit4:junit4] 2> 234587 T459 C276 P12293 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> C272_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:28895_ogmsi%2Ftl, base_url=http://127.0.0.1:28895/ogmsi/tl}
[junit4:junit4] 2> 234587 T495 C272 P28895 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 234603 T476 C273 P27106 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5afde945; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5afde945; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 234604 T459 C276 P12293 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d854b15; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d854b15; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 234604 T495 C272 P28895 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6164799f; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6164799f; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 234605 T459 C276 P12293 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 234604 T476 C273 P27106 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 234606 T495 C272 P28895 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 234609 T459 C276 P12293 oass.SolrIndexSearcher.<init> Opening Searcher@49812388 main
[junit4:junit4] 2> 234609 T476 C273 P27106 oass.SolrIndexSearcher.<init> Opening Searcher@31fa6d39 main
[junit4:junit4] 2> 234609 T495 C272 P28895 oass.SolrIndexSearcher.<init> Opening Searcher@2db8b002 main
[junit4:junit4] 2> 234609 T459 C276 P12293 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 234610 T471 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@49812388 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 234610 T495 C272 P28895 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 234610 T476 C273 P27106 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 234612 T507 oasc.JmxMonitoredMap.put WARN Failed to register info bean: fieldValueCache javax.management.InstanceAlreadyExistsException: solr/collection1:type=fieldValueCache,id=org.apache.solr.search.FastLRUCache
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:513)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4] 2> at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:303)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1791)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.access$000(SolrCore.java:132)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1668)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2>
[junit4:junit4] 2> 234612 T459 C276 P12293 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 25
[junit4:junit4] 2> 234611 T488 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@31fa6d39 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 234613 T507 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2db8b002 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 234614 T476 C273 P27106 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 27
[junit4:junit4] 2> 234614 T495 C272 P28895 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 27
[junit4:junit4] 2> 234615 T445 C274 P30366 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 93
[junit4:junit4] 2> 234617 T411 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 234622 T447 C274 P30366 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 234626 T460 C276 P12293 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 234630 T477 C273 P27106 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 234633 T496 C272 P28895 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 236653 T423 C275 P36610 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1436541495101358080)} 0 15
[junit4:junit4] 2> 236673 T495 C272 P28895 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={update.distrib=FROMLEADER&_version_=-1436541495114989568&update.from=http://127.0.0.1:30366/ogmsi/tl/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1436541495114989568)} 0 2
[junit4:junit4] 2> 236673 T476 C273 P27106 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={update.distrib=FROMLEADER&_version_=-1436541495114989568&update.from=http://127.0.0.1:30366/ogmsi/tl/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1436541495114989568)} 0 2
[junit4:junit4] 2> 236673 T459 C276 P12293 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={update.distrib=FROMLEADER&_version_=-1436541495114989568&update.from=http://127.0.0.1:30366/ogmsi/tl/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1436541495114989568)} 0 2
[junit4:junit4] 2> 236675 T445 C274 P30366 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1436541495114989568)} 0 19
[junit4:junit4] 2> 236683 T423 C275 P36610 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={wt=javabin&version=2} {add=[0 (1436541495138058240)]} 0 5
[junit4:junit4] 2> 236710 T459 C276 P12293 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={distrib.from=http://127.0.0.1:30366/ogmsi/tl/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1436541495146446848)]} 0 4
[junit4:junit4] 2> 236710 T476 C273 P27106 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={distrib.from=http://127.0.0.1:30366/ogmsi/tl/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1436541495146446848)]} 0 4
[junit4:junit4] 2> 236710 T495 C272 P28895 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={distrib.from=http://127.0.0.1:30366/ogmsi/tl/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1436541495146446848)]} 0 4
[junit4:junit4] 2> 236712 T445 C274 P30366 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={wt=javabin&version=2} {add=[0 (1436541495146446848)]} 0 26
[junit4:junit4] 2> 236718 T423 C275 P36610 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={wt=javabin&version=2} {add=[1 (1436541495177904128)]} 0 2
[junit4:junit4] 2> 236727 T495 C272 P28895 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={distrib.from=http://127.0.0.1:30366/ogmsi/tl/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1436541495183147008)]} 0 2
[junit4:junit4] 2> 236727 T459 C276 P12293 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={distrib.from=http://127.0.0.1:30366/ogmsi/tl/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1436541495183147008)]} 0 2
[junit4:junit4] 2> 236727 T476 C273 P27106 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={distrib.from=http://127.0.0.1:30366/ogmsi/tl/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1436541495183147008)]} 0 2
[junit4:junit4] 2> 236729 T445 C274 P30366 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={wt=javabin&version=2} {add=[1 (1436541495183147008)]} 0 8
[junit4:junit4] 2> 236734 T423 C275 P36610 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={wt=javabin&version=2} {add=[2 (1436541495194681344)]} 0 2
[junit4:junit4] 2> 236738 T445 C274 P30366 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:12293/ogmsi/tl/collection1/]
[junit4:junit4] 2> 236738 T445 C274 P30366 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:12293/ogmsi/tl/collection1/ against:[http://127.0.0.1:12293/ogmsi/tl/collection1/] result:true
[junit4:junit4] 2> 236739 T445 C274 P30366 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:27106/ogmsi/tl/collection1/ against:[http://127.0.0.1:12293/ogmsi/tl/collection1/] result:false
[junit4:junit4] 2> 236739 T445 C274 P30366 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:28895/ogmsi/tl/collection1/ against:[http://127.0.0.1:12293/ogmsi/tl/collection1/] result:false
[junit4:junit4] 2> 236745 T476 C273 P27106 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={distrib.from=http://127.0.0.1:30366/ogmsi/tl/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:12293/ogmsi/tl/collection1/&wt=javabin&version=2} {add=[2 (1436541495200972800)]} 0 2
[junit4:junit4] 2> 236745 T495 C272 P28895 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={distrib.from=http://127.0.0.1:30366/ogmsi/tl/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:12293/ogmsi/tl/collection1/&wt=javabin&version=2} {add=[2 (1436541495200972800)]} 0 2
[junit4:junit4] 2> 236746 T445 C274 P30366 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={test.distrib.skip.servers=http://127.0.0.1:12293/ogmsi/tl/collection1/&wt=javabin&version=2} {add=[2 (1436541495200972800)]} 0 9
[junit4:junit4] 2> 236751 T423 C275 P36610 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={wt=javabin&version=2} {add=[3 (1436541495212507136)]} 0 2
[junit4:junit4] 2> 236756 T445 C274 P30366 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:12293/ogmsi/tl/collection1/, http://127.0.0.1:27106/ogmsi/tl/collection1/]
[junit4:junit4] 2> 236756 T445 C274 P30366 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:12293/ogmsi/tl/collection1/ against:[http://127.0.0.1:12293/ogmsi/tl/collection1/, http://127.0.0.1:27106/ogmsi/tl/collection1/] result:true
[junit4:junit4] 2> 236757 T445 C274 P30366 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:27106/ogmsi/tl/collection1/ against:[http://127.0.0.1:12293/ogmsi/tl/collection1/, http://127.0.0.1:27106/ogmsi/tl/collection1/] result:true
[junit4:junit4] 2> 236757 T445 C274 P30366 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:28895/ogmsi/tl/collection1/ against:[http://127.0.0.1:12293/ogmsi/tl/collection1/, http://127.0.0.1:27106/ogmsi/tl/collection1/] result:false
[junit4:junit4] 2> 236763 T495 C272 P28895 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={distrib.from=http://127.0.0.1:30366/ogmsi/tl/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:12293/ogmsi/tl/collection1/&wt=javabin&version=2} {add=[3 (1436541495220895744)]} 0 2
[junit4:junit4] 2> 236764 T445 C274 P30366 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={test.distrib.skip.servers=http://127.0.0.1:12293/ogmsi/tl/collection1/&test.distrib.skip.servers=http://127.0.0.1:27106/ogmsi/tl/collection1/&wt=javabin&version=2} {add=[3 (1436541495220895744)]} 0 9
[junit4:junit4] 2> 236767 T423 C275 P36610 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 236861 T423 C275 P36610 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e3eb0d8; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e3eb0d8; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 236862 T423 C275 P36610 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 236880 T423 C275 P36610 oass.SolrIndexSearcher.<init> Opening Searcher@7e416dcc main
[junit4:junit4] 2> 236881 T423 C275 P36610 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 236881 T437 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7e416dcc main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 236882 T423 C275 P36610 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 115
[junit4:junit4] 2> 236887 T445 C274 P30366 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 237025 T445 C274 P30366 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@12b56813; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> 237026 T445 C274 P30366 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 237043 T445 C274 P30366 oass.SolrIndexSearcher.<init> Opening Searcher@3b362c2 main
[junit4:junit4] 2> 237044 T445 C274 P30366 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 237044 T455 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3b362c2 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 237045 T445 C274 P30366 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:12293/ogmsi/tl/collection1/, StdNode: http://127.0.0.1:27106/ogmsi/tl/collection1/, StdNode: http://127.0.0.1:28895/ogmsi/tl/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 237049 T459 C276 P12293 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 237049 T495 C272 P28895 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 237049 T476 C273 P27106 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 237203 T459 C276 P12293 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d854b15; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d854b15; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 237203 T495 C272 P28895 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6164799f; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6164799f; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> 237203 T459 C276 P12293 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 237204 T495 C272 P28895 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 237206 T476 C273 P27106 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5afde945; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1369992701963/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5afde945; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 237207 T476 C273 P27106 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 237219 T495 C272 P28895 oass.SolrIndexSearcher.<init> Opening Searcher@6e0092d1 main
[junit4:junit4] 2> 237219 T459 C276 P12293 oass.SolrIndexSearcher.<init> Opening Searcher@67a70fc2 main
[junit4:junit4] 2> 237220 T495 C272 P28895 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 237220 T459 C276 P12293 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 237220 T471 oasc.JmxMonitoredMap.put WARN Failed to register info bean: searcher javax.management.InstanceAlreadyExistsException: solr/collection1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:513)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4] 2> at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:299)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1791)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.access$000(SolrCore.java:132)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1668)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2>
[junit4:junit4] 2> 237221 T507 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6e0092d1 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 237221 T471 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@67a70fc2 main{StandardDirectoryReader(segments_4:3:nrt _0(5.0):C2)}
[junit4:junit4] 2> 237222 T495 C272 P28895 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 173
[junit4:junit4] 2> 237222 T459 C276 P12293 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 173
[junit4:junit4] 2> 237226 T476 C273 P27106 oass.SolrIndexSearcher.<init> Opening Searcher@70a8e788 main
[junit4:junit4] 2> 237227 T476 C273 P27106 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 237227 T488 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@70a8e788 main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):C3)}
[junit4:junit4] 2> 237228 T476 C273 P27106 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 179
[junit4:junit4] 2> 237229 T445 C274 P30366 oasup.LogUpdateProcessor.finish [collection1] webapp=/ogmsi/tl path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 342
[junit4:junit4] 2> 237229 T411 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 237230 T411 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 237231 T411 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 237235 T447 C274 P30366 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 237239 T460 C276 P12293 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=1
[junit4:junit4] 2> 237242 T477 C273 P27106 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=3 status=0 QTime=0
[junit4:junit4] 2> 237246 T496 C272 P28895 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 237251 T411 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 237270 T478 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4] 2> 237271 T478 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 237282 T449 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4] 2> 237282 T449 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:30366/ogmsi/tl/collection1/
[junit4:junit4] 2> 237282 T449 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:30366/ogmsi/tl START replicas=[http://127.0.0.1:12293/ogmsi/tl/collection1/, http://127.0.0.1:27106/ogmsi/tl/collection1/, http://127.0.0.1:28895/ogmsi/tl/collection1/] nUpdates=100
[junit4:junit4] 2> 237293 T497 C272 P28895 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=8
[junit4:junit4] 2> 237293 T479 C273 P27106 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=8
[junit4:junit4] 2> 237293 T461 C276 P12293 oasc.SolrCore.execute [collection1] webapp=/ogmsi/tl path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=8
[junit4:junit4] 2> 237294 T449 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:30366/ogmsi/tl Received 5 versions from 127.0.0.1:28895/ogmsi/tl/collection1/
[junit4:junit4] 2> 237295 T449 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:30366/ogmsi/tl Our versions are newer. ourLowThreshold=1436541495114989568 otherHigh=1436541495200972800
[junit4:junit4] 2> 237295 T449 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:30366/ogmsi/tl Received 4 versions from 127.0.0.1:27106/ogmsi/tl/collection1/
[junit4:junit4] 2> 237295 T449 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:30366/ogmsi/tl Our versions ar
[...truncated too long message...]
] 2> 2072242 T2570 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 2072242 T2570 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 2073070 T2512 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 2073070 T2512 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2073073 T2512 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@240045bf
[junit4:junit4] 2> 2073079 T2512 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=10,adds=10,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=52,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 2073080 T2512 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 2073081 T2512 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 2073081 T2512 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> ASYNC NEW_CORE C771 name=collection1 org.apache.solr.core.SolrCore@240045bf url=http://127.0.0.1:47489/collection1 node=127.0.0.1:47489_ C771_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:47489_, base_url=http://127.0.0.1:47489, leader=true}
[junit4:junit4] 2> 2073083 T2512 C771 P47489 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5f575e9c lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e2e58d9),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5f575e9c lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e2e58d9),segFN=segments_5,generation=5}
[junit4:junit4] 2> 2073084 T2512 C771 P47489 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 2073086 T2512 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 2073087 T2512 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 2073087 T2512 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ShardSplitTest-1369994529299/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ShardSplitTest-1369994529299/jetty4;done=false>>]
[junit4:junit4] 2> 2073088 T2512 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ShardSplitTest-1369994529299/jetty4
[junit4:junit4] 2> 2073088 T2512 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ShardSplitTest-1369994529299/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ShardSplitTest-1369994529299/jetty4/index;done=false>>]
[junit4:junit4] 2> 2073089 T2512 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ShardSplitTest-1369994529299/jetty4/index
[junit4:junit4] 2> 2073090 T2634 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89783961472663564-127.0.0.1:47489_-n_0000000004) am no longer a leader.
[junit4:junit4] 2> 2073092 T2512 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 2073164 T2512 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 2073165 T2512 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:13473 13473
[junit4:junit4] 2> 2073316 T2586 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 2073455 T2586 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 2073455 T2586 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 2073456 T2512 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 56166
[junit4:junit4] 2> !!!! WARNING: best effort to remove /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ShardSplitTest-1369994529299 FAILED !!!!!
[junit4:junit4] 2> 2073457 T2512 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 2073458 T2512 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:13473 13473
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ShardSplitTest -Dtests.method=testDistribSearch -Dtests.seed=2FC1E187373C5811 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=fr -Dtests.timezone=Asia/Yekaterinburg -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 35.9s J0 | ShardSplitTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:38300 returned non ok status:500, message:Server Error
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([2FC1E187373C5811:AE276F9F4063382D]:0)
[junit4:junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
[junit4:junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4] > at org.apache.solr.cloud.ShardSplitTest.splitShard(ShardSplitTest.java:208)
[junit4:junit4] > at org.apache.solr.cloud.ShardSplitTest.doTest(ShardSplitTest.java:133)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 2073478 T2512 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 35876 T2511 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4] 2> 2074392 T2603 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 2074392 T2603 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2074392 T2603 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=fr, timezone=Asia/Yekaterinburg
[junit4:junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=1,free=139892240,total=459276288
[junit4:junit4] 2> NOTE: All tests run in this JVM: [SolrCoreTest, AutoCommitTest, LegacyHTMLStripCharFilterTest, TestStressRecovery, MinimalSchemaTest, TestConfig, ZkControllerTest, DistributedTermsComponentTest, AlternateDirectoryTest, TestJoin, SpatialFilterTest, FileBasedSpellCheckerTest, IndexSchemaTest, DocumentBuilderTest, TestFunctionQuery, TestSearchPerf, TestQuerySenderListener, TestSolrQueryParser, SOLR749Test, BinaryUpdateRequestHandlerTest, TestRemoteStreaming, TestQueryUtils, OverseerCollectionProcessorTest, TestFieldCollectionResource, TestFieldTypeCollectionResource, TestSweetSpotSimilarityFactory, QueryElevationComponentTest, SyncSliceTest, TestManagedSchema, DistributedQueryElevationComponentTest, DateFieldTest, SimpleFacetsTest, ShowFileRequestHandlerTest, CoreAdminHandlerTest, TestLRUCache, LukeRequestHandlerTest, ClusterStateUpdateTest, TestFuzzyAnalyzedSuggestions, LeaderElectionTest, SoftAutoCommitTest, TestSurroundQueryParser, DocValuesTest, TestSchemaResource, TestPerFieldSimilarity, TestIBSimilarityFactory, SliceStateUpdateTest, TestSort, SchemaVersionSpecificBehaviorTest, PrimUtilsTest, CircularListTest, BasicFunctionalityTest, TestClassNameShortening, TestWriterPerf, JsonLoaderTest, TestCoreContainer, TestSolrDeletionPolicy2, AliasIntegrationTest, RegexBoostProcessorTest, TestGroupingSearch, PreAnalyzedUpdateProcessorTest, TestRandomDVFaceting, BasicDistributedZkTest, ZkCLITest, RecoveryZkTest, ChaosMonkeyShardSplitTest, BadComponentTest, UnloadDistributedZkTest, ShardSplitTest]
[junit4:junit4] Completed on J0 in 36.92s, 1 test, 1 error <<< FAILURES!
[...truncated 25 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:380: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:360: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/solr/common-build.xml:437: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:1240: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-trunk-Java7/lucene/common-build.xml:884: There were test failures: 295 suites, 1232 tests, 1 error, 1 failure, 13 ignored (7 assumptions)
Total time: 57 minutes 21 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure