You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/05/27 03:55:33 UTC
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0-ea-b89) - Build #
5847 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/5847/
Java: 32bit/jdk1.8.0-ea-b89 -server -XX:+UseParallelGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeyShardSplitTest.testDistribSearch
Error Message:
shard2 is not consistent. Got 49 from http://127.0.0.1:50876/uv/b/collection1lastClient and got 48 from http://127.0.0.1:48513/uv/b/collection1
Stack Trace:
java.lang.AssertionError: shard2 is not consistent. Got 49 from http://127.0.0.1:50876/uv/b/collection1lastClient and got 48 from http://127.0.0.1:48513/uv/b/collection1
at __randomizedtesting.SeedInfo.seed([DC9515E394AF7C1C:5D739BFBE3F01C20]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:110)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:491)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9603 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeyShardSplitTest
[junit4:junit4] 2> 222024 T1239 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /uv/b
[junit4:junit4] 2> 222029 T1239 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeyShardSplitTest-1369618924882
[junit4:junit4] 2> 222029 T1239 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 222030 T1240 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 222130 T1239 oasc.ZkTestServer.run start zk server on port:47398
[junit4:junit4] 2> 222131 T1239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 222165 T1246 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@117e3fa name:ZooKeeperConnection Watcher:127.0.0.1:47398 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 222165 T1239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 222165 T1239 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 222174 T1239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 222176 T1248 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d00a28 name:ZooKeeperConnection Watcher:127.0.0.1:47398/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 222177 T1239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 222177 T1239 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 222184 T1239 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 222189 T1239 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 222194 T1239 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 222200 T1239 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 222200 T1239 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 222209 T1239 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 222210 T1239 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 222216 T1239 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 222217 T1239 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 222222 T1239 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 222223 T1239 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 222228 T1239 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 222229 T1239 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 222235 T1239 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 222236 T1239 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 222244 T1239 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 222245 T1239 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 222250 T1239 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 222250 T1239 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 222256 T1239 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 222256 T1239 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 222333 T1239 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 222336 T1239 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42207
[junit4:junit4] 2> 222337 T1239 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 222337 T1239 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 222337 T1239 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369618925116
[junit4:junit4] 2> 222337 T1239 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369618925116/solr.xml
[junit4:junit4] 2> 222338 T1239 oasc.CoreContainer.<init> New CoreContainer 29404046
[junit4:junit4] 2> 222338 T1239 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369618925116/'
[junit4:junit4] 2> 222338 T1239 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369618925116/'
[junit4:junit4] 2> 222373 T1239 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 222373 T1239 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 222374 T1239 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 222374 T1239 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 222374 T1239 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 222375 T1239 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 222375 T1239 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 222375 T1239 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 222375 T1239 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 222376 T1239 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 222378 T1239 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 222379 T1239 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47398/solr
[junit4:junit4] 2> 222379 T1239 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 222380 T1239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 222381 T1259 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19460f8 name:ZooKeeperConnection Watcher:127.0.0.1:47398 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 222381 T1239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 222384 T1239 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 222387 T1239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 222388 T1261 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1128d5 name:ZooKeeperConnection Watcher:127.0.0.1:47398/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 222389 T1239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 222391 T1239 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 222397 T1239 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 222401 T1239 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 222404 T1239 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42207_uv%2Fb
[junit4:junit4] 2> 222406 T1239 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42207_uv%2Fb
[junit4:junit4] 2> 222411 T1239 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 222419 T1239 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 222424 T1239 oasc.Overseer.start Overseer (id=89759345861328899-127.0.0.1:42207_uv%2Fb-n_0000000000) starting
[junit4:junit4] 2> 222431 T1239 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 222438 T1263 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 222439 T1239 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 222444 T1239 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 222448 T1239 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 222454 T1262 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 222458 T1264 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369618925116/collection1
[junit4:junit4] 2> 222458 T1264 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 222459 T1264 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 222459 T1264 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 222462 T1264 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369618925116/collection1/'
[junit4:junit4] 2> 222463 T1264 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369618925116/collection1/lib/README' to classloader
[junit4:junit4] 2> 222463 T1264 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369618925116/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 222506 T1264 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 222535 T1264 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 222537 T1264 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 222542 T1264 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 222788 T1264 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 222790 T1264 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 222791 T1264 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 222795 T1264 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 222835 T1264 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 222836 T1264 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-controljetty-1369618925116/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/control/data/
[junit4:junit4] 2> 222836 T1264 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1fd96dd
[junit4:junit4] 2> 222837 T1264 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 222837 T1264 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/control/data
[junit4:junit4] 2> 222838 T1264 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/control/data/index/
[junit4:junit4] 2> 222838 T1264 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 222839 T1264 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/control/data/index
[junit4:junit4] 2> 222841 T1264 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1c837ae lockFactory=org.apache.lucene.store.NativeFSLockFactory@18084d2),segFN=segments_1,generation=1}
[junit4:junit4] 2> 222842 T1264 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 222844 T1264 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 222845 T1264 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 222846 T1264 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 222846 T1264 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 222847 T1264 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 222847 T1264 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 222848 T1264 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 222848 T1264 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 222849 T1264 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 222852 T1264 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 222856 T1264 oass.SolrIndexSearcher.<init> Opening Searcher@1583ad7 main
[junit4:junit4] 2> 222856 T1264 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/control/data/tlog
[junit4:junit4] 2> 222857 T1264 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 222857 T1264 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 222862 T1265 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1583ad7 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 222863 T1264 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 222863 T1264 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 223961 T1262 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 223963 T1262 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42207/uv/b",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "node_name":"127.0.0.1:42207_uv%2Fb",
[junit4:junit4] 2> "shard_state":"active"}
[junit4:junit4] 2> 223963 T1262 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 223963 T1262 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 223971 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 224867 T1264 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 224867 T1264 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42207/uv/b collection:control_collection shard:shard1
[junit4:junit4] 2> 224869 T1264 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 224883 T1264 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 224887 T1264 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 224887 T1264 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 224887 T1264 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42207/uv/b/collection1/
[junit4:junit4] 2> 224888 T1264 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 224888 T1264 oasc.SyncStrategy.syncToMe http://127.0.0.1:42207/uv/b/collection1/ has no replicas
[junit4:junit4] 2> 224888 T1264 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42207/uv/b/collection1/
[junit4:junit4] 2> 224889 T1264 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 225480 T1262 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 225493 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 225503 T1264 oasc.ZkController.register We are http://127.0.0.1:42207/uv/b/collection1/ and leader is http://127.0.0.1:42207/uv/b/collection1/
[junit4:junit4] 2> 225504 T1264 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42207/uv/b
[junit4:junit4] 2> 225504 T1264 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 225504 T1264 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 225504 T1264 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 225507 T1264 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 225509 T1239 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 225510 T1239 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 225511 T1239 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 225517 T1239 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 225519 T1239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 225521 T1268 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5cfa7a name:ZooKeeperConnection Watcher:127.0.0.1:47398/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 225522 T1239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 225525 T1239 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 225531 T1239 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 225604 T1239 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 225610 T1239 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50876
[junit4:junit4] 2> 225611 T1239 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 225611 T1239 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 225611 T1239 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369618928384
[junit4:junit4] 2> 225612 T1239 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369618928384/solr.xml
[junit4:junit4] 2> 225612 T1239 oasc.CoreContainer.<init> New CoreContainer 1983287
[junit4:junit4] 2> 225612 T1239 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369618928384/'
[junit4:junit4] 2> 225612 T1239 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369618928384/'
[junit4:junit4] 2> 225648 T1239 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 225649 T1239 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 225649 T1239 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 225649 T1239 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 225650 T1239 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 225650 T1239 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 225650 T1239 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 225650 T1239 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 225651 T1239 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 225651 T1239 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 225653 T1239 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 225654 T1239 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47398/solr
[junit4:junit4] 2> 225654 T1239 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 225655 T1239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 225656 T1279 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1402ecf name:ZooKeeperConnection Watcher:127.0.0.1:47398 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 225657 T1239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 225659 T1239 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 225661 T1239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 225663 T1281 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@111e8b4 name:ZooKeeperConnection Watcher:127.0.0.1:47398/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 225663 T1239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 225667 T1239 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 226670 T1239 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50876_uv%2Fb
[junit4:junit4] 2> 226673 T1239 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50876_uv%2Fb
[junit4:junit4] 2> 226676 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 226678 T1268 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 226678 T1281 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 226681 T1261 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 226690 T1282 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369618928384/collection1
[junit4:junit4] 2> 226690 T1282 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 226691 T1282 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 226692 T1282 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 226694 T1282 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369618928384/collection1/'
[junit4:junit4] 2> 226695 T1282 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369618928384/collection1/lib/README' to classloader
[junit4:junit4] 2> 226695 T1282 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369618928384/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 226740 T1282 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 226768 T1282 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 226770 T1282 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 226773 T1282 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 227004 T1262 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 227005 T1262 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:42207_uv%2Fb_collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42207/uv/b",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "node_name":"127.0.0.1:42207_uv%2Fb",
[junit4:junit4] 2> "shard_state":"active"}
[junit4:junit4] 2> 227010 T1281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 227010 T1268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 227010 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 227022 T1282 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 227024 T1282 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 227025 T1282 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 227029 T1282 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 227071 T1282 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 227071 T1282 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty1-1369618928384/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty1/
[junit4:junit4] 2> 227072 T1282 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1fd96dd
[junit4:junit4] 2> 227072 T1282 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 227073 T1282 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty1
[junit4:junit4] 2> 227073 T1282 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty1/index/
[junit4:junit4] 2> 227074 T1282 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 227074 T1282 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty1/index
[junit4:junit4] 2> 227076 T1282 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1608aef lockFactory=org.apache.lucene.store.NativeFSLockFactory@269577),segFN=segments_1,generation=1}
[junit4:junit4] 2> 227077 T1282 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 227080 T1282 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 227080 T1282 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 227081 T1282 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 227082 T1282 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 227083 T1282 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 227083 T1282 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 227083 T1282 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 227084 T1282 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 227085 T1282 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 227088 T1282 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 227092 T1282 oass.SolrIndexSearcher.<init> Opening Searcher@1929b0 main
[junit4:junit4] 2> 227092 T1282 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty1/tlog
[junit4:junit4] 2> 227093 T1282 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 227093 T1282 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 227099 T1283 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1929b0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 227102 T1282 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 227102 T1282 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 228517 T1262 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 228518 T1262 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50876/uv/b",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "node_name":"127.0.0.1:50876_uv%2Fb",
[junit4:junit4] 2> "shard_state":"active"}
[junit4:junit4] 2> 228519 T1262 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 228519 T1262 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 228527 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 228527 T1281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 228527 T1268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 229105 T1282 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 229105 T1282 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50876/uv/b collection:collection1 shard:shard2
[junit4:junit4] 2> 229106 T1282 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 229120 T1282 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 229125 T1282 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 229125 T1282 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 229126 T1282 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50876/uv/b/collection1/
[junit4:junit4] 2> 229126 T1282 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 229126 T1282 oasc.SyncStrategy.syncToMe http://127.0.0.1:50876/uv/b/collection1/ has no replicas
[junit4:junit4] 2> 229126 T1282 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50876/uv/b/collection1/
[junit4:junit4] 2> 229126 T1282 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 230036 T1262 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 230053 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 230053 T1281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 230053 T1268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 230087 T1282 oasc.ZkController.register We are http://127.0.0.1:50876/uv/b/collection1/ and leader is http://127.0.0.1:50876/uv/b/collection1/
[junit4:junit4] 2> 230087 T1282 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50876/uv/b
[junit4:junit4] 2> 230087 T1282 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 230087 T1282 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 230088 T1282 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 230091 T1282 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 230093 T1239 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 230094 T1239 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 230095 T1239 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 230161 T1239 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 230163 T1239 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51008
[junit4:junit4] 2> 230163 T1239 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 230164 T1239 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 230164 T1239 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369618932953
[junit4:junit4] 2> 230164 T1239 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369618932953/solr.xml
[junit4:junit4] 2> 230164 T1239 oasc.CoreContainer.<init> New CoreContainer 14139451
[junit4:junit4] 2> 230165 T1239 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369618932953/'
[junit4:junit4] 2> 230165 T1239 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369618932953/'
[junit4:junit4] 2> 230199 T1239 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 230199 T1239 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 230200 T1239 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 230200 T1239 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 230200 T1239 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 230200 T1239 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 230201 T1239 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 230201 T1239 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 230201 T1239 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 230201 T1239 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 230204 T1239 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 230204 T1239 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47398/solr
[junit4:junit4] 2> 230204 T1239 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 230205 T1239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 230207 T1295 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18331d7 name:ZooKeeperConnection Watcher:127.0.0.1:47398 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 230207 T1239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 230210 T1239 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 230215 T1239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 230217 T1297 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e1ef55 name:ZooKeeperConnection Watcher:127.0.0.1:47398/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 230218 T1239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 230226 T1239 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 231232 T1239 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51008_uv%2Fb
[junit4:junit4] 2> 231235 T1239 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51008_uv%2Fb
[junit4:junit4] 2> 231240 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 231240 T1281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 231241 T1268 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 231241 T1297 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 231241 T1268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 231244 T1261 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 231244 T1281 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 231251 T1298 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369618932953/collection1
[junit4:junit4] 2> 231251 T1298 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 231252 T1298 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 231253 T1298 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 231255 T1298 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369618932953/collection1/'
[junit4:junit4] 2> 231256 T1298 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369618932953/collection1/lib/README' to classloader
[junit4:junit4] 2> 231257 T1298 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369618932953/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 231282 T1298 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 231307 T1298 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 231309 T1298 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 231312 T1298 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 231566 T1262 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 231567 T1262 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:50876_uv%2Fb_collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50876/uv/b",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "node_name":"127.0.0.1:50876_uv%2Fb",
[junit4:junit4] 2> "shard_state":"active"}
[junit4:junit4] 2> 231572 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 231572 T1281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 231572 T1268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 231572 T1297 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 231582 T1298 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 231584 T1298 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 231585 T1298 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 231588 T1298 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 231632 T1298 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 231632 T1298 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty2-1369618932953/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty2/
[junit4:junit4] 2> 231633 T1298 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1fd96dd
[junit4:junit4] 2> 231634 T1298 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 231635 T1298 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty2
[junit4:junit4] 2> 231635 T1298 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty2/index/
[junit4:junit4] 2> 231636 T1298 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 231637 T1298 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty2/index
[junit4:junit4] 2> 231640 T1298 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@188d5df lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d478dc),segFN=segments_1,generation=1}
[junit4:junit4] 2> 231641 T1298 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 231645 T1298 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 231646 T1298 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 231647 T1298 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 231648 T1298 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 231648 T1298 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 231649 T1298 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 231649 T1298 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 231649 T1298 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 231650 T1298 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 231653 T1298 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 231657 T1298 oass.SolrIndexSearcher.<init> Opening Searcher@3f6f34 main
[junit4:junit4] 2> 231658 T1298 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty2/tlog
[junit4:junit4] 2> 231659 T1298 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 231659 T1298 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 231666 T1299 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3f6f34 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 231668 T1298 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 231668 T1298 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 233078 T1262 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 233079 T1262 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51008/uv/b",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "node_name":"127.0.0.1:51008_uv%2Fb",
[junit4:junit4] 2> "shard_state":"active"}
[junit4:junit4] 2> 233080 T1262 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 233080 T1262 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 233088 T1297 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 233088 T1281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 233088 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 233088 T1268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 233671 T1298 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 233671 T1298 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51008/uv/b collection:collection1 shard:shard1
[junit4:junit4] 2> 233673 T1298 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 233683 T1298 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 233687 T1298 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 233687 T1298 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 233687 T1298 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51008/uv/b/collection1/
[junit4:junit4] 2> 233688 T1298 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 233688 T1298 oasc.SyncStrategy.syncToMe http://127.0.0.1:51008/uv/b/collection1/ has no replicas
[junit4:junit4] 2> 233688 T1298 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51008/uv/b/collection1/
[junit4:junit4] 2> 233689 T1298 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 234598 T1262 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 234614 T1268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 234615 T1297 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 234615 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 234615 T1281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 234651 T1298 oasc.ZkController.register We are http://127.0.0.1:51008/uv/b/collection1/ and leader is http://127.0.0.1:51008/uv/b/collection1/
[junit4:junit4] 2> 234651 T1298 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51008/uv/b
[junit4:junit4] 2> 234652 T1298 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 234652 T1298 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 234652 T1298 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 234655 T1298 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 234657 T1239 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 234658 T1239 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 234659 T1239 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 234734 T1239 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 234737 T1239 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48513
[junit4:junit4] 2> 234737 T1239 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 234737 T1239 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 234738 T1239 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369618937518
[junit4:junit4] 2> 234738 T1239 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369618937518/solr.xml
[junit4:junit4] 2> 234738 T1239 oasc.CoreContainer.<init> New CoreContainer 32272550
[junit4:junit4] 2> 234739 T1239 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369618937518/'
[junit4:junit4] 2> 234739 T1239 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369618937518/'
[junit4:junit4] 2> 234773 T1239 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 234773 T1239 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 234773 T1239 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 234774 T1239 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 234774 T1239 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 234774 T1239 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 234774 T1239 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 234775 T1239 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 234775 T1239 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 234775 T1239 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 234778 T1239 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 234778 T1239 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47398/solr
[junit4:junit4] 2> 234778 T1239 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 234779 T1239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 234781 T1311 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c36dda name:ZooKeeperConnection Watcher:127.0.0.1:47398 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 234782 T1239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 234785 T1239 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 234788 T1239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 234790 T1313 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e07a1 name:ZooKeeperConnection Watcher:127.0.0.1:47398/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 234790 T1239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 234798 T1239 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 235804 T1239 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48513_uv%2Fb
[junit4:junit4] 2> 235807 T1239 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48513_uv%2Fb
[junit4:junit4] 2> 235812 T1281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 235813 T1268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 235813 T1313 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 235813 T1261 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 235814 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 235814 T1297 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 235814 T1297 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 235816 T1281 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 235817 T1268 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 235824 T1314 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369618937518/collection1
[junit4:junit4] 2> 235824 T1314 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 235825 T1314 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 235826 T1314 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 235828 T1314 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369618937518/collection1/'
[junit4:junit4] 2> 235829 T1314 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369618937518/collection1/lib/README' to classloader
[junit4:junit4] 2> 235829 T1314 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369618937518/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 235869 T1314 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 235894 T1314 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 235896 T1314 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 235899 T1314 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 236126 T1262 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 236127 T1262 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:51008_uv%2Fb_collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51008/uv/b",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "node_name":"127.0.0.1:51008_uv%2Fb",
[junit4:junit4] 2> "shard_state":"active"}
[junit4:junit4] 2> 236132 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 236132 T1268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 236132 T1297 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 236132 T1313 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 236132 T1281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 236148 T1314 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 236150 T1314 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 236151 T1314 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 236155 T1314 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 236193 T1314 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 236194 T1314 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty3-1369618937518/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty3/
[junit4:junit4] 2> 236194 T1314 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1fd96dd
[junit4:junit4] 2> 236195 T1314 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 236196 T1314 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty3
[junit4:junit4] 2> 236196 T1314 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty3/index/
[junit4:junit4] 2> 236196 T1314 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 236197 T1314 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty3/index
[junit4:junit4] 2> 236199 T1314 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@e90231 lockFactory=org.apache.lucene.store.NativeFSLockFactory@ef6fd2),segFN=segments_1,generation=1}
[junit4:junit4] 2> 236199 T1314 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 236202 T1314 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 236203 T1314 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 236204 T1314 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 236204 T1314 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 236205 T1314 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 236205 T1314 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 236206 T1314 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 236206 T1314 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 236206 T1314 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 236209 T1314 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 236211 T1314 oass.SolrIndexSearcher.<init> Opening Searcher@318d33 main
[junit4:junit4] 2> 236212 T1314 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty3/tlog
[junit4:junit4] 2> 236212 T1314 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 236212 T1314 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 236218 T1315 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@318d33 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 236220 T1314 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 236220 T1314 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 237639 T1262 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 237641 T1262 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48513/uv/b",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "node_name":"127.0.0.1:48513_uv%2Fb",
[junit4:junit4] 2> "shard_state":"active"}
[junit4:junit4] 2> 237641 T1262 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 237641 T1262 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 237649 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 237650 T1297 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 237650 T1268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 237649 T1313 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 237649 T1281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 238223 T1314 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 238223 T1314 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48513/uv/b collection:collection1 shard:shard2
[junit4:junit4] 2> 238230 T1314 oasc.ZkController.register We are http://127.0.0.1:48513/uv/b/collection1/ and leader is http://127.0.0.1:50876/uv/b/collection1/
[junit4:junit4] 2> 238230 T1314 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48513/uv/b
[junit4:junit4] 2> 238231 T1314 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 238231 T1314 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C326 name=collection1 org.apache.solr.core.SolrCore@1bef75e url=http://127.0.0.1:48513/uv/b/collection1 node=127.0.0.1:48513_uv%2Fb C326_STATE=coll:collection1 core:collection1 props:{state=down, collection=collection1, core=collection1, base_url=http://127.0.0.1:48513/uv/b, node_name=127.0.0.1:48513_uv%2Fb}
[junit4:junit4] 2> 238231 T1316 C326 P48513 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 238232 T1316 C326 P48513 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 238233 T1316 C326 P48513 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 238233 T1314 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 238233 T1316 C326 P48513 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 238235 T1239 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 238236 T1239 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 238235 T1316 C326 P48513 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 238236 T1239 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 238242 T1277 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:48513_uv%2Fb_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 238315 T1239 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 238317 T1239 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60747
[junit4:junit4] 2> 238318 T1239 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 238319 T1239 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 238319 T1239 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369618941093
[junit4:junit4] 2> 238320 T1239 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369618941093/solr.xml
[junit4:junit4] 2> 238320 T1239 oasc.CoreContainer.<init> New CoreContainer 8477731
[junit4:junit4] 2> 238321 T1239 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369618941093/'
[junit4:junit4] 2> 238322 T1239 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369618941093/'
[junit4:junit4] 2> 238358 T1239 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 238358 T1239 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 238359 T1239 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 238359 T1239 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 238359 T1239 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 238359 T1239 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 238360 T1239 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 238360 T1239 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 238360 T1239 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 238360 T1239 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 238363 T1239 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 238363 T1239 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47398/solr
[junit4:junit4] 2> 238364 T1239 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 238364 T1239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 238366 T1328 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c67af name:ZooKeeperConnection Watcher:127.0.0.1:47398 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 238366 T1239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 238369 T1239 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 238375 T1239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 238377 T1330 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d27737 name:ZooKeeperConnection Watcher:127.0.0.1:47398/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 238377 T1239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 238384 T1239 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 239159 T1262 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 239161 T1262 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48513_uv%2Fb_collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48513/uv/b",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "node_name":"127.0.0.1:48513_uv%2Fb",
[junit4:junit4] 2> "shard_state":"active"}
[junit4:junit4] 2> 239170 T1297 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 239170 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 239170 T1313 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 239170 T1268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 239170 T1281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 239170 T1330 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 239243 T1277 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:48513_uv%2Fb_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 239243 T1277 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={state=recovering&nodeName=127.0.0.1:48513_uv%252Fb&action=PREPRECOVERY&onlyIfLeader=true&checkLive=true&core=collection1&coreNodeName=127.0.0.1:48513_uv%252Fb_collection1&wt=javabin&version=2} status=0 QTime=1001
[junit4:junit4] 2> 239389 T1239 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60747_uv%2Fb
[junit4:junit4] 2> 239391 T1239 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60747_uv%2Fb
[junit4:junit4] 2> 239395 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 239396 T1281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 239397 T1297 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 239397 T1330 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 239397 T1297 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 239397 T1268 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 239397 T1313 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 239397 T1330 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 239398 T1313 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 239398 T1268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 239400 T1261 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 239401 T1281 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 239409 T1331 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369618941093/collection1
[junit4:junit4] 2> 239409 T1331 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 239410 T1331 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 239411 T1331 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 239413 T1331 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369618941093/collection1/'
[junit4:junit4] 2> 239414 T1331 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369618941093/collection1/lib/README' to classloader
[junit4:junit4] 2> 239414 T1331 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369618941093/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 239447 T1331 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 239472 T1331 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 239475 T1331 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 239477 T1331 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 239733 T1331 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 239735 T1331 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 239736 T1331 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 239740 T1331 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 239783 T1331 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 239783 T1331 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-jetty4-1369618941093/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty4/
[junit4:junit4] 2> 239784 T1331 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1fd96dd
[junit4:junit4] 2> 239784 T1331 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 239785 T1331 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty4
[junit4:junit4] 2> 239785 T1331 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty4/index/
[junit4:junit4] 2> 239786 T1331 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 239786 T1331 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty4/index
[junit4:junit4] 2> 239789 T1331 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3073cc lockFactory=org.apache.lucene.store.NativeFSLockFactory@774590),segFN=segments_1,generation=1}
[junit4:junit4] 2> 239789 T1331 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 239792 T1331 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 239792 T1331 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 239793 T1331 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 239794 T1331 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 239795 T1331 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 239795 T1331 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 239795 T1331 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 239796 T1331 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 239797 T1331 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 239800 T1331 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 239804 T1331 oass.SolrIndexSearcher.<init> Opening Searcher@1cede16 main
[junit4:junit4] 2> 239804 T1331 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty4/tlog
[junit4:junit4] 2> 239805 T1331 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 239805 T1331 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 239809 T1332 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1cede16 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 239810 T1331 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 239810 T1331 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 240680 T1262 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 240682 T1262 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60747/uv/b",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "node_name":"127.0.0.1:60747_uv%2Fb",
[junit4:junit4] 2> "shard_state":"active"}
[junit4:junit4] 2> 240682 T1262 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 240683 T1262 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 240691 T1281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 240691 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 240692 T1330 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 240691 T1297 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 240691 T1313 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 240691 T1268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 240812 T1331 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 240812 T1331 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60747/uv/b collection:collection1 shard:shard1
[junit4:junit4] 2> 240818 T1331 oasc.ZkController.register We are http://127.0.0.1:60747/uv/b/collection1/ and leader is http://127.0.0.1:51008/uv/b/collection1/
[junit4:junit4] 2> 240819 T1331 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60747/uv/b
[junit4:junit4] 2> 240819 T1331 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 240819 T1331 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C327 name=collection1 org.apache.solr.core.SolrCore@10f6043 url=http://127.0.0.1:60747/uv/b/collection1 node=127.0.0.1:60747_uv%2Fb C327_STATE=coll:collection1 core:collection1 props:{state=down, collection=collection1, core=collection1, base_url=http://127.0.0.1:60747/uv/b, node_name=127.0.0.1:60747_uv%2Fb}
[junit4:junit4] 2> 240820 T1333 C327 P60747 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 240821 T1333 C327 P60747 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 240821 T1333 C327 P60747 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 240821 T1333 C327 P60747 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 240821 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 240823 T1333 C327 P60747 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 240824 T1239 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 240825 T1239 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 240825 T1239 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 240830 T1291 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:60747_uv%2Fb_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 240830 T1239 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 240832 T1239 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 240833 T1239 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 240835 T1239 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C328 name=collection1 org.apache.solr.core.SolrCore@1bef75e url=http://127.0.0.1:48513/uv/b/collection1 node=127.0.0.1:48513_uv%2Fb C328_STATE=coll:collection1 core:collection1 props:{state=recovering, collection=collection1, shard=shard2, core=collection1, base_url=http://127.0.0.1:48513/uv/b, node_name=127.0.0.1:48513_uv%2Fb}
[junit4:junit4] 2> 241245 T1316 C328 P48513 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:50876/uv/b/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 241245 T1316 C328 P48513 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:48513/uv/b START replicas=[http://127.0.0.1:50876/uv/b/collection1/] nUpdates=100
[junit4:junit4] 2> 241246 T1316 C328 P48513 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 241246 T1316 C328 P48513 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 241246 T1316 C328 P48513 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 241246 T1316 C328 P48513 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 241247 T1316 C328 P48513 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 241247 T1316 C328 P48513 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:50876/uv/b/collection1/. core=collection1
[junit4:junit4] 2> 241247 T1316 C328 P48513 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C329 name=collection1 org.apache.solr.core.SolrCore@1c84e24 url=http://127.0.0.1:50876/uv/b/collection1 node=127.0.0.1:50876_uv%2Fb C329_STATE=coll:collection1 core:collection1 props:{state=active, collection=collection1, shard=shard2, core=collection1, base_url=http://127.0.0.1:50876/uv/b, node_name=127.0.0.1:50876_uv%2Fb, leader=true}
[junit4:junit4] 2> 241252 T1274 C329 P50876 oasc.SolrCore.execute [collection1] webapp=/uv/b path=/get params={getVersions=100&distrib=false&qt=/get&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 241257 T1276 C329 P50876 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 241259 T1276 C329 P50876 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1608aef lockFactory=org.apache.lucene.store.NativeFSLockFactory@269577),segFN=segments_1,generation=1}
[junit4:junit4] 2> 241260 T1276 C329 P50876 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 241261 T1276 C329 P50876 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1608aef lockFactory=org.apache.lucene.store.NativeFSLockFactory@269577),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1608aef lockFactory=org.apache.lucene.store.NativeFSLockFactory@269577),segFN=segments_2,generation=2}
[junit4:junit4] 2> 241261 T1276 C329 P50876 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 241262 T1276 C329 P50876 oass.SolrIndexSearcher.<init> Opening Searcher@39847f realtime
[junit4:junit4] 2> 241263 T1276 C329 P50876 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 241263 T1276 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={waitSearcher=true&openSearcher=false&commit_end_point=true&softCommit=false&wt=javabin&version=2&commit=true} {commit=} 0 6
[junit4:junit4] 2> 241264 T1316 C328 P48513 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 241264 T1316 C328 P48513 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 241270 T1275 C329 P50876 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 241271 T1275 C329 P50876 oasc.SolrCore.execute [collection1] webapp=/uv/b path=/replication params={qt=/replication&wt=javabin&command=indexversion&version=2} status=0 QTime=5
[junit4:junit4] 2> 241272 T1316 C328 P48513 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 241272 T1316 C328 P48513 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 241272 T1316 C328 P48513 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 241275 T1274 C329 P50876 oasc.SolrCore.execute [collection1] webapp=/uv/b path=/replication params={generation=2&qt=/replication&wt=javabin&command=filelist&version=2} status=0 QTime=1
[junit4:junit4] 2> 241275 T1316 C328 P48513 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 241277 T1316 C328 P48513 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty3/index.20130527034224129
[junit4:junit4] 2> 241277 T1316 C328 P48513 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@12613d1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@180e5a) fullCopy=false
[junit4:junit4] 2> 241280 T1276 C329 P50876 oasc.SolrCore.execute [collection1] webapp=/uv/b path=/replication params={generation=2&file=segments_2&qt=/replication&wt=filestream&command=filecontent&checksum=true} status=0 QTime=0
[junit4:junit4] 2> 241282 T1316 C328 P48513 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 241283 T1316 C328 P48513 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 241283 T1316 C328 P48513 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 241285 T1316 C328 P48513 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@e90231 lockFactory=org.apache.lucene.store.NativeFSLockFactory@ef6fd2),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@e90231 lockFactory=org.apache.lucene.store.NativeFSLockFactory@ef6fd2),segFN=segments_2,generation=2}
[junit4:junit4] 2> 241285 T1316 C328 P48513 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 241285 T1316 C328 P48513 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 241286 T1316 C328 P48513 oass.SolrIndexSearcher.<init> Opening Searcher@a694a2 main
[junit4:junit4] 2> 241287 T1315 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a694a2 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 241288 T1316 C328 P48513 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty3/index.20130527034224129 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty3/index.20130527034224129;done=true>>]
[junit4:junit4] 2> 241288 T1316 C328 P48513 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty3/index.20130527034224129
[junit4:junit4] 2> 241289 T1316 C328 P48513 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty3/index.20130527034224129
[junit4:junit4] 2> 241289 T1316 C328 P48513 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 241289 T1316 C328 P48513 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 241289 T1316 C328 P48513 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 241290 T1316 C328 P48513 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 241292 T1316 C328 P48513 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 241838 T1239 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 242201 T1262 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 242202 T1262 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:60747_uv%2Fb_collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60747/uv/b",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "node_name":"127.0.0.1:60747_uv%2Fb",
[junit4:junit4] 2> "shard_state":"active"}
[junit4:junit4] 2> 242210 T1262 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48513_uv%2Fb_collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48513/uv/b",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "node_name":"127.0.0.1:48513_uv%2Fb",
[junit4:junit4] 2> "shard_state":"active"}
[junit4:junit4] 2> 242219 T1330 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 242219 T1281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 242219 T1297 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 242219 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 242219 T1313 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 242219 T1268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 242831 T1291 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:60747_uv%2Fb_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 242831 T1291 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={state=recovering&nodeName=127.0.0.1:60747_uv%252Fb&action=PREPRECOVERY&onlyIfLeader=true&checkLive=true&core=collection1&coreNodeName=127.0.0.1:60747_uv%252Fb_collection1&wt=javabin&version=2} status=0 QTime=2001
[junit4:junit4] 2> 242841 T1239 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 243844 T1239 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C327_STATE=coll:collection1 core:collection1 props:{state=recovering, collection=collection1, shard=shard1, core=collection1, base_url=http://127.0.0.1:60747/uv/b, node_name=127.0.0.1:60747_uv%2Fb}
[junit4:junit4] 2> 244833 T1333 C327 P60747 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:51008/uv/b/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 244833 T1333 C327 P60747 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:60747/uv/b START replicas=[http://127.0.0.1:51008/uv/b/collection1/] nUpdates=100
[junit4:junit4] 2> 244834 T1333 C327 P60747 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 244834 T1333 C327 P60747 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 244834 T1333 C327 P60747 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 244834 T1333 C327 P60747 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 244835 T1333 C327 P60747 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 244835 T1333 C327 P60747 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:51008/uv/b/collection1/. core=collection1
[junit4:junit4] 2> 244835 T1333 C327 P60747 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C330 name=collection1 org.apache.solr.core.SolrCore@1979d45 url=http://127.0.0.1:51008/uv/b/collection1 node=127.0.0.1:51008_uv%2Fb C330_STATE=coll:collection1 core:collection1 props:{state=active, collection=collection1, shard=shard1, core=collection1, base_url=http://127.0.0.1:51008/uv/b, node_name=127.0.0.1:51008_uv%2Fb, leader=true}
[junit4:junit4] 2> 244843 T1291 C330 P51008 oasc.SolrCore.execute [collection1] webapp=/uv/b path=/get params={getVersions=100&distrib=false&qt=/get&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 244847 T1239 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 244849 T1293 C330 P51008 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 244851 T1293 C330 P51008 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@188d5df lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d478dc),segFN=segments_1,generation=1}
[junit4:junit4] 2> 244851 T1293 C330 P51008 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 244852 T1293 C330 P51008 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@188d5df lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d478dc),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@188d5df lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d478dc),segFN=segments_2,generation=2}
[junit4:junit4] 2> 244853 T1293 C330 P51008 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 244853 T1293 C330 P51008 oass.SolrIndexSearcher.<init> Opening Searcher@985a55 realtime
[junit4:junit4] 2> 244854 T1293 C330 P51008 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 244854 T1293 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={waitSearcher=true&openSearcher=false&commit_end_point=true&softCommit=false&wt=javabin&version=2&commit=true} {commit=} 0 6
[junit4:junit4] 2> 244856 T1333 C327 P60747 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 244856 T1333 C327 P60747 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 244858 T1292 C330 P51008 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 244859 T1292 C330 P51008 oasc.SolrCore.execute [collection1] webapp=/uv/b path=/replication params={qt=/replication&wt=javabin&command=indexversion&version=2} status=0 QTime=2
[junit4:junit4] 2> 244859 T1333 C327 P60747 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 244860 T1333 C327 P60747 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 244860 T1333 C327 P60747 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 244862 T1291 C330 P51008 oasc.SolrCore.execute [collection1] webapp=/uv/b path=/replication params={generation=2&qt=/replication&wt=javabin&command=filelist&version=2} status=0 QTime=0
[junit4:junit4] 2> 244863 T1333 C327 P60747 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 244864 T1333 C327 P60747 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty4/index.20130527034227717
[junit4:junit4] 2> 244865 T1333 C327 P60747 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@1cb32cd lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c0b2b) fullCopy=false
[junit4:junit4] 2> 244868 T1293 C330 P51008 oasc.SolrCore.execute [collection1] webapp=/uv/b path=/replication params={generation=2&file=segments_2&qt=/replication&wt=filestream&command=filecontent&checksum=true} status=0 QTime=1
[junit4:junit4] 2> 244869 T1333 C327 P60747 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 244870 T1333 C327 P60747 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 244871 T1333 C327 P60747 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 244872 T1333 C327 P60747 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3073cc lockFactory=org.apache.lucene.store.NativeFSLockFactory@774590),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3073cc lockFactory=org.apache.lucene.store.NativeFSLockFactory@774590),segFN=segments_2,generation=2}
[junit4:junit4] 2> 244873 T1333 C327 P60747 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 244873 T1333 C327 P60747 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 244873 T1333 C327 P60747 oass.SolrIndexSearcher.<init> Opening Searcher@ae3d38 main
[junit4:junit4] 2> 244875 T1332 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ae3d38 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 244875 T1333 C327 P60747 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty4/index.20130527034227717 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty4/index.20130527034227717;done=true>>]
[junit4:junit4] 2> 244876 T1333 C327 P60747 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty4/index.20130527034227717
[junit4:junit4] 2> 244876 T1333 C327 P60747 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty4/index.20130527034227717
[junit4:junit4] 2> 244876 T1333 C327 P60747 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 244877 T1333 C327 P60747 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 244877 T1333 C327 P60747 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 244877 T1333 C327 P60747 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 244879 T1333 C327 P60747 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 245235 T1262 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 245237 T1262 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:60747_uv%2Fb_collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60747/uv/b",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "node_name":"127.0.0.1:60747_uv%2Fb",
[junit4:junit4] 2> "shard_state":"active"}
[junit4:junit4] 2> 245245 T1297 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 245245 T1313 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 245245 T1261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 245245 T1281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 245245 T1268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 245245 T1330 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 245851 T1239 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 245853 T1239 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C331 name=collection1 org.apache.solr.core.SolrCore@106283 url=http://127.0.0.1:42207/uv/b/collection1 node=127.0.0.1:42207_uv%2Fb C331_STATE=coll:control_collection core:collection1 props:{state=active, collection=control_collection, shard=shard1, core=collection1, base_url=http://127.0.0.1:42207/uv/b, node_name=127.0.0.1:42207_uv%2Fb, leader=true}
[junit4:junit4] 2> 245861 T1257 C331 P42207 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 245863 T1257 C331 P42207 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1c837ae lockFactory=org.apache.lucene.store.NativeFSLockFactory@18084d2),segFN=segments_1,generation=1}
[junit4:junit4] 2> 245864 T1257 C331 P42207 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 245865 T1257 C331 P42207 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1c837ae lockFactory=org.apache.lucene.store.NativeFSLockFactory@18084d2),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1c837ae lockFactory=org.apache.lucene.store.NativeFSLockFactory@18084d2),segFN=segments_2,generation=2}
[junit4:junit4] 2> 245866 T1257 C331 P42207 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 245867 T1257 C331 P42207 oass.SolrIndexSearcher.<init> Opening Searcher@1eb47ba main
[junit4:junit4] 2> 245867 T1257 C331 P42207 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 245868 T1265 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1eb47ba main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 245869 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={waitSearcher=true&softCommit=false&wt=javabin&version=2&commit=true} {commit=} 0 8
[junit4:junit4] 2> 245875 T1293 C330 P51008 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 245875 T1293 C330 P51008 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@188d5df lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d478dc),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@188d5df lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d478dc),segFN=segments_3,generation=3}
[junit4:junit4] 2> 245876 T1293 C330 P51008 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 245877 T1293 C330 P51008 oass.SolrIndexSearcher.<init> Opening Searcher@53c97 main
[junit4:junit4] 2> 245877 T1293 C330 P51008 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 245878 T1299 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@53c97 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 245879 T1293 C330 P51008 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:60747/uv/b/collection1/, StdNode: http://127.0.0.1:50876/uv/b/collection1/, StdNode: http://127.0.0.1:48513/uv/b/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 245884 T1274 C329 P50876 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 245885 T1274 C329 P50876 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1608aef lockFactory=org.apache.lucene.store.NativeFSLockFactory@269577),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1608aef lockFactory=org.apache.lucene.store.NativeFSLockFactory@269577),segFN=segments_3,generation=3}
[junit4:junit4] 2> 245886 T1274 C329 P50876 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 245887 T1274 C329 P50876 oass.SolrIndexSearcher.<init> Opening Searcher@1f74d98 main
[junit4:junit4] 2> 245888 T1274 C329 P50876 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 245889 T1283 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f74d98 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 245889 T1274 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={waitSearcher=true&expungeDeletes=false&commit_end_point=true&softCommit=false&wt=javabin&version=2&commit=true} {commit=} 0 5
[junit4:junit4] 2> C328_STATE=coll:collection1 core:collection1 props:{state=active, collection=collection1, shard=shard2, core=collection1, base_url=http://127.0.0.1:48513/uv/b, node_name=127.0.0.1:48513_uv%2Fb}
[junit4:junit4] 2> 245890 T1309 C328 P48513 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 245891 T1309 C328 P48513 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@e90231 lockFactory=org.apache.lucene.store.NativeFSLockFactory@ef6fd2),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@e90231 lockFactory=org.apache.lucene.store.NativeFSLockFactory@ef6fd2),segFN=segments_3,generation=3}
[junit4:junit4] 2> 245892 T1309 C328 P48513 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> C327_STATE=coll:collection1 core:collection1 props:{state=active, collection=collection1, shard=shard1, core=collection1, base_url=http://127.0.0.1:60747/uv/b, node_name=127.0.0.1:60747_uv%2Fb}
[junit4:junit4] 2> 245892 T1325 C327 P60747 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 245893 T1309 C328 P48513 oass.SolrIndexSearcher.<init> Opening Searcher@18212f3 main
[junit4:junit4] 2> 245893 T1325 C327 P60747 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3073cc lockFactory=org.apache.lucene.store.NativeFSLockFactory@774590),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3073cc lockFactory=org.apache.lucene.store.NativeFSLockFactory@774590),segFN=segments_3,generation=3}
[junit4:junit4] 2> 245894 T1309 C328 P48513 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 245894 T1325 C327 P60747 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 245895 T1315 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@18212f3 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 245896 T1325 C327 P60747 oass.SolrIndexSearcher.<init> Opening Searcher@365b67 main
[junit4:junit4] 2> 245896 T1309 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={waitSearcher=true&expungeDeletes=false&commit_end_point=true&softCommit=false&wt=javabin&version=2&commit=true} {commit=} 0 6
[junit4:junit4] 2> 245896 T1325 C327 P60747 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 245897 T1332 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@365b67 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 245898 T1325 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={waitSearcher=true&expungeDeletes=false&commit_end_point=true&softCommit=false&wt=javabin&version=2&commit=true} {commit=} 0 6
[junit4:junit4] 2> 245899 T1293 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={waitSearcher=true&softCommit=false&wt=javabin&version=2&commit=true} {commit=} 0 25
[junit4:junit4] 2> 245901 T1239 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 245904 T1276 C329 P50876 oasc.SolrCore.execute [collection1] webapp=/uv/b path=/select params={distrib=false&q=*:*&wt=javabin&rows=0&version=2&tests=checkShardConsistency} hits=0 status=0 QTime=0
[junit4:junit4] 2> 245907 T1306 C328 P48513 oasc.SolrCore.execute [collection1] webapp=/uv/b path=/select params={distrib=false&q=*:*&wt=javabin&rows=0&version=2&tests=checkShardConsistency} hits=0 status=0 QTime=1
[junit4:junit4] 2> 245909 T1292 C330 P51008 oasc.SolrCore.execute [collection1] webapp=/uv/b path=/select params={distrib=false&q=*:*&wt=javabin&rows=0&version=2&tests=checkShardConsistency} hits=0 status=0 QTime=0
[junit4:junit4] 2> 245912 T1326 C327 P60747 oasc.SolrCore.execute [collection1] webapp=/uv/b path=/select params={distrib=false&q=*:*&wt=javabin&rows=0&version=2&tests=checkShardConsistency} hits=0 status=0 QTime=1
[junit4:junit4] 2> 247917 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1436149560921554944)} 0 2
[junit4:junit4] 2> 247924 T1326 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={_version_=-1436149560925749248&update.distrib=FROMLEADER&update.from=http://127.0.0.1:51008/uv/b/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1436149560925749248)} 0 1
[junit4:junit4] 2> 247929 T1306 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={_version_=-1436149560928894976&update.distrib=FROMLEADER&update.from=http://127.0.0.1:50876/uv/b/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1436149560928894976)} 0 1
[junit4:junit4] 2> 247930 T1274 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1436149560928894976)} 0 7
[junit4:junit4] 2> 247931 T1290 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1436149560925749248)} 0 12
[junit4:junit4] 2> 247935 T1254 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[0 (1436149560940429312)]} 0 1
[junit4:junit4] 2> 247943 T1323 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1436149560944623616)]} 0 2
[junit4:junit4] 2> 247944 T1291 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[0 (1436149560944623616)]} 0 6
[junit4:junit4] 2> 247947 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[1 (1436149560954060800)]} 0 0
[junit4:junit4] 2> 247956 T1324 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1436149560960352256)]} 0 0
[junit4:junit4] 2> 247957 T1293 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1436149560960352256)]} 0 4
[junit4:junit4] 2> 247958 T1277 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[1]} 0 9
[junit4:junit4] 2> 247961 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[2 (1436149560968740864)]} 0 0
[junit4:junit4] 2> 247968 T1307 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[2 (1436149560971886592)]} 0 1
[junit4:junit4] 2> 247969 T1275 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[2 (1436149560971886592)]} 0 5
[junit4:junit4] 2> 247972 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[3 (1436149560979226624)]} 0 1
[junit4:junit4] 2> 247978 T1308 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[3 (1436149560983420928)]} 0 1
[junit4:junit4] 2> 247979 T1276 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[3 (1436149560983420928)]} 0 5
[junit4:junit4] 2> 247982 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[4 (1436149560990760960)]} 0 0
[junit4:junit4] 2> 247988 T1325 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[4 (1436149560993906688)]} 0 0
[junit4:junit4] 2> 247989 T1292 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[4 (1436149560993906688)]} 0 4
[junit4:junit4] 2> 247992 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[5 (1436149561001246720)]} 0 0
[junit4:junit4] 2> 247998 T1309 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[5 (1436149561004392448)]} 0 1
[junit4:junit4] 2> 247999 T1274 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[5 (1436149561004392448)]} 0 4
[junit4:junit4] 2> 248002 T1254 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[6 (1436149561011732480)]} 0 1
[junit4:junit4] 2> 248008 T1306 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[6 (1436149561014878208)]} 0 1
[junit4:junit4] 2> 248008 T1277 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[6 (1436149561014878208)]} 0 3
[junit4:junit4] 2> 248011 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[7 (1436149561021169664)]} 0 0
[junit4:junit4] 2> 248019 T1307 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[7 (1436149561027461120)]} 0 0
[junit4:junit4] 2> 248020 T1275 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[7 (1436149561027461120)]} 0 4
[junit4:junit4] 2> 248021 T1290 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[7]} 0 7
[junit4:junit4] 2> 248024 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[8 (1436149561034801152)]} 0 0
[junit4:junit4] 2> 248032 T1326 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[8 (1436149561040044032)]} 0 0
[junit4:junit4] 2> 248033 T1291 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[8 (1436149561040044032)]} 0 4
[junit4:junit4] 2> 248034 T1276 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[8]} 0 7
[junit4:junit4] 2> 248037 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[9 (1436149561048432640)]} 0 0
[junit4:junit4] 2> 248045 T1308 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[9 (1436149561053675520)]} 0 0
[junit4:junit4] 2> 248046 T1274 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[9 (1436149561053675520)]} 0 4
[junit4:junit4] 2> 248047 T1293 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[9]} 0 7
[junit4:junit4] 2> 248050 T1254 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[10 (1436149561062064128)]} 0 0
[junit4:junit4] 2> 248058 T1323 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10 (1436149561068355584)]} 0 0
[junit4:junit4] 2> 248059 T1292 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[10 (1436149561068355584)]} 0 4
[junit4:junit4] 2> 248060 T1277 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[10]} 0 7
[junit4:junit4] 2> 248063 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[11 (1436149561075695616)]} 0 0
[junit4:junit4] 2> 248069 T1324 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[11 (1436149561078841344)]} 0 0
[junit4:junit4] 2> 248070 T1290 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[11 (1436149561078841344)]} 0 4
[junit4:junit4] 2> 248073 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[12 (1436149561086181376)]} 0 0
[junit4:junit4] 2> 248081 T1325 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1436149561092472832)]} 0 0
[junit4:junit4] 2> 248082 T1291 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[12 (1436149561092472832)]} 0 4
[junit4:junit4] 2> 248083 T1275 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[12]} 0 7
[junit4:junit4] 2> 248087 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[13 (1436149561099812864)]} 0 1
[junit4:junit4] 2> 248095 T1326 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1436149561106104320)]} 0 1
[junit4:junit4] 2> 248095 T1293 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[13 (1436149561106104320)]} 0 4
[junit4:junit4] 2> 248096 T1276 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[13]} 0 7
[junit4:junit4] 2> 248099 T1254 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[14 (1436149561113444352)]} 0 0
[junit4:junit4] 2> 248105 T1323 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1436149561116590080)]} 0 1
[junit4:junit4] 2> 248106 T1292 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[14 (1436149561116590080)]} 0 4
[junit4:junit4] 2> 248109 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[15 (1436149561123930112)]} 0 1
[junit4:junit4] 2> 248117 T1324 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1436149561129172992)]} 0 0
[junit4:junit4] 2> 248118 T1290 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[15 (1436149561129172992)]} 0 4
[junit4:junit4] 2> 248119 T1274 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[15]} 0 7
[junit4:junit4] 2> 248122 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[16 (1436149561137561600)]} 0 0
[junit4:junit4] 2> 248128 T1325 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1436149561140707328)]} 0 0
[junit4:junit4] 2> 248129 T1291 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[16 (1436149561140707328)]} 0 4
[junit4:junit4] 2> 248132 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[17 (1436149561148047360)]} 0 0
[junit4:junit4] 2> 248141 T1309 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1436149561153290240)]} 0 1
[junit4:junit4] 2> 248141 T1277 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[17 (1436149561153290240)]} 0 4
[junit4:junit4] 2> 248142 T1293 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[17]} 0 7
[junit4:junit4] 2> 248145 T1254 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[18 (1436149561161678848)]} 0 0
[junit4:junit4] 2> 248151 T1306 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1436149561164824576)]} 0 0
[junit4:junit4] 2> 248152 T1275 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[18 (1436149561164824576)]} 0 4
[junit4:junit4] 2> 248155 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[19 (1436149561172164608)]} 0 0
[junit4:junit4] 2> 248161 T1307 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1436149561175310336)]} 0 0
[junit4:junit4] 2> 248162 T1276 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[19 (1436149561175310336)]} 0 4
[junit4:junit4] 2> 248165 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[20 (1436149561182650368)]} 0 1
[junit4:junit4] 2> 248171 T1326 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1436149561185796096)]} 0 1
[junit4:junit4] 2> 248172 T1292 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[20 (1436149561185796096)]} 0 4
[junit4:junit4] 2> 248175 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[21 (1436149561192087552)]} 0 1
[junit4:junit4] 2> 248183 T1308 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1436149561198379008)]} 0 1
[junit4:junit4] 2> 248184 T1274 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[21 (1436149561198379008)]} 0 4
[junit4:junit4] 2> 248185 T1290 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[21]} 0 8
[junit4:junit4] 2> 248188 T1254 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[22 (1436149561205719040)]} 0 1
[junit4:junit4] 2> 248196 T1323 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1436149561212010496)]} 0 1
[junit4:junit4] 2> 248197 T1291 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[22 (1436149561212010496)]} 0 4
[junit4:junit4] 2> 248197 T1277 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[22]} 0 7
[junit4:junit4] 2> 248201 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[23 (1436149561219350528)]} 0 1
[junit4:junit4] 2> 248206 T1309 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1436149561223544832)]} 0 0
[junit4:junit4] 2> 248207 T1275 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[23 (1436149561223544832)]} 0 4
[junit4:junit4] 2> 248210 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[24 (1436149561229836288)]} 0 0
[junit4:junit4] 2> 248218 T1324 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1436149561236127744)]} 0 0
[junit4:junit4] 2> 248219 T1293 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[24 (1436149561236127744)]} 0 4
[junit4:junit4] 2> 248220 T1276 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[24]} 0 7
[junit4:junit4] 2> 248223 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[25 (1436149561243467776)]} 0 0
[junit4:junit4] 2> 248232 T1306 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1436149561249759232)]} 0 1
[junit4:junit4] 2> 248232 T1274 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[25 (1436149561249759232)]} 0 4
[junit4:junit4] 2> 248233 T1292 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[25]} 0 7
[junit4:junit4] 2> 248236 T1254 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[26 (1436149561257099264)]} 0 0
[junit4:junit4] 2> 248245 T1307 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1436149561263390720)]} 0 1
[junit4:junit4] 2> 248245 T1277 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[26 (1436149561263390720)]} 0 4
[junit4:junit4] 2> 248246 T1290 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[26]} 0 7
[junit4:junit4] 2> 248250 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[27 (1436149561270730752)]} 0 1
[junit4:junit4] 2> 248255 T1325 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1436149561273876480)]} 0 0
[junit4:junit4] 2> 248256 T1291 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[27 (1436149561273876480)]} 0 4
[junit4:junit4] 2> 248259 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[28 (1436149561281216512)]} 0 0
[junit4:junit4] 2> 248267 T1326 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1436149561287507968)]} 0 0
[junit4:junit4] 2> 248268 T1293 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[28 (1436149561287507968)]} 0 4
[junit4:junit4] 2> 248269 T1275 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[28]} 0 7
[junit4:junit4] 2> 248272 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[29 (1436149561294848000)]} 0 0
[junit4:junit4] 2> 248281 T1308 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1436149561301139456)]} 0 1
[junit4:junit4] 2> 248281 T1276 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[29 (1436149561301139456)]} 0 4
[junit4:junit4] 2> 248282 T1292 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[29]} 0 7
[junit4:junit4] 2> 248285 T1254 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[30 (1436149561308479488)]} 0 0
[junit4:junit4] 2> 248293 T1309 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1436149561314770944)]} 0 0
[junit4:junit4] 2> 248294 T1274 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[30 (1436149561314770944)]} 0 4
[junit4:junit4] 2> 248295 T1290 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[30]} 0 7
[junit4:junit4] 2> 248298 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[31 (1436149561322110976)]} 0 0
[junit4:junit4] 2> 248307 T1306 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1436149561328402432)]} 0 1
[junit4:junit4] 2> 248308 T1277 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[31 (1436149561328402432)]} 0 5
[junit4:junit4] 2> 248309 T1291 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[31]} 0 8
[junit4:junit4] 2> 248312 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[32 (1436149561336791040)]} 0 1
[junit4:junit4] 2> 248320 T1324 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1436149561342033920)]} 0 1
[junit4:junit4] 2> 248321 T1293 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[32 (1436149561342033920)]} 0 4
[junit4:junit4] 2> 248321 T1276 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[32]} 0 6
[junit4:junit4] 2> 248325 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[33 (1436149561349373952)]} 0 1
[junit4:junit4] 2> 248331 T1309 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1436149561353568256)]} 0 1
[junit4:junit4] 2> 248331 T1274 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[33 (1436149561353568256)]} 0 4
[junit4:junit4] 2> 248335 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[34 (1436149561359859712)]} 0 1
[junit4:junit4] 2> 248343 T1326 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1436149561366151168)]} 0 1
[junit4:junit4] 2> 248344 T1290 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[34 (1436149561366151168)]} 0 4
[junit4:junit4] 2> 248344 T1275 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[34]} 0 7
[junit4:junit4] 2> 248348 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[35 (1436149561373491200)]} 0 1
[junit4:junit4] 2> 248356 T1323 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1436149561379782656)]} 0 1
[junit4:junit4] 2> 248357 T1291 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[35 (1436149561379782656)]} 0 4
[junit4:junit4] 2> 248358 T1276 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[35]} 0 8
[junit4:junit4] 2> 248361 T1254 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[36 (1436149561387122688)]} 0 1
[junit4:junit4] 2> 248369 T1325 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1436149561393414144)]} 0 1
[junit4:junit4] 2> 248370 T1293 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[36 (1436149561393414144)]} 0 4
[junit4:junit4] 2> 248370 T1277 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[36]} 0 7
[junit4:junit4] 2> 248374 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[37 (1436149561400754176)]} 0 1
[junit4:junit4] 2> 248380 T1307 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1436149561404948480)]} 0 0
[junit4:junit4] 2> 248380 T1274 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[37 (1436149561404948480)]} 0 4
[junit4:junit4] 2> 248384 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[38 (1436149561411239936)]} 0 1
[junit4:junit4] 2> 248392 T1324 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[38 (1436149561417531392)]} 0 1
[junit4:junit4] 2> 248393 T1292 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[38 (1436149561417531392)]} 0 4
[junit4:junit4] 2> 248393 T1275 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[38]} 0 7
[junit4:junit4] 2> 248397 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[39 (1436149561424871424)]} 0 1
[junit4:junit4] 2> 248402 T1306 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[39 (1436149561428017152)]} 0 0
[junit4:junit4] 2> 248403 T1276 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[39 (1436149561428017152)]} 0 4
[junit4:junit4] 2> 248407 T1254 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[40 (1436149561435357184)]} 0 1
[junit4:junit4] 2> 248415 T1326 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[40 (1436149561441648640)]} 0 1
[junit4:junit4] 2> 248416 T1290 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[40 (1436149561441648640)]} 0 4
[junit4:junit4] 2> 248417 T1277 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[40]} 0 8
[junit4:junit4] 2> 248420 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[41 (1436149561448988672)]} 0 1
[junit4:junit4] 2> 248426 T1323 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[41 (1436149561453182976)]} 0 1
[junit4:junit4] 2> 248426 T1291 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[41 (1436149561453182976)]} 0 4
[junit4:junit4] 2> 248430 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[42 (1436149561459474432)]} 0 1
[junit4:junit4] 2> 248435 T1325 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[42 (1436149561463668736)]} 0 0
[junit4:junit4] 2> 248436 T1293 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[42 (1436149561463668736)]} 0 4
[junit4:junit4] 2> 248439 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[43 (1436149561469960192)]} 0 0
[junit4:junit4] 2> 248445 T1324 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[43 (1436149561473105920)]} 0 0
[junit4:junit4] 2> 248446 T1292 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[43 (1436149561473105920)]} 0 4
[junit4:junit4] 2> 248449 T1254 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[44 (1436149561480445952)]} 0 0
[junit4:junit4] 2> 248457 T1326 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[44 (1436149561486737408)]} 0 0
[junit4:junit4] 2> 248458 T1290 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[44 (1436149561486737408)]} 0 4
[junit4:junit4] 2> 248459 T1274 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[44]} 0 7
[junit4:junit4] 2> 248463 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[45 (1436149561494077440)]} 0 0
[junit4:junit4] 2> 248468 T1308 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[45 (1436149561497223168)]} 0 0
[junit4:junit4] 2> 248469 T1275 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[45 (1436149561497223168)]} 0 4
[junit4:junit4] 2> 248472 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[46 (1436149561504563200)]} 0 0
[junit4:junit4] 2> 248478 T1323 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[46 (1436149561507708928)]} 0 0
[junit4:junit4] 2> 248479 T1291 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[46 (1436149561507708928)]} 0 4
[junit4:junit4] 2> 248482 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[47 (1436149561515048960)]} 0 0
[junit4:junit4] 2> 248490 T1325 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[47 (1436149561520291840)]} 0 0
[junit4:junit4] 2> 248491 T1293 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[47 (1436149561520291840)]} 0 4
[junit4:junit4] 2> 248492 T1276 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[47]} 0 7
[junit4:junit4] 2> 248495 T1254 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[48 (1436149561528680448)]} 0 1
[junit4:junit4] 2> 248503 T1324 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[48 (1436149561533923328)]} 0 1
[junit4:junit4] 2> 248504 T1292 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[48 (1436149561533923328)]} 0 4
[junit4:junit4] 2> 248505 T1277 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[48]} 0 7
[junit4:junit4] 2> 248508 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[49 (1436149561541263360)]} 0 1
[junit4:junit4] 2> 248516 T1326 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[49 (1436149561547554816)]} 0 1
[junit4:junit4] 2> 248517 T1290 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[49 (1436149561547554816)]} 0 4
[junit4:junit4] 2> 248518 T1274 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[49]} 0 8
[junit4:junit4] 2> 248521 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[50 (1436149561555943424)]} 0 1
[junit4:junit4] 2> 248529 T1323 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[50 (1436149561561186304)]} 0 0
[junit4:junit4] 2> 248530 T1291 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[50 (1436149561561186304)]} 0 4
[junit4:junit4] 2> 248531 T1275 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[50]} 0 7
[junit4:junit4] 2> 248534 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[51 (1436149561569574912)]} 0 0
[junit4:junit4] 2> 248540 T1325 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[51 (1436149561572720640)]} 0 1
[junit4:junit4] 2> 248541 T1293 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[51 (1436149561572720640)]} 0 4
[junit4:junit4] 2> 248544 T1254 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[52 (1436149561580060672)]} 0 0
[junit4:junit4] 2> 248552 T1324 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[52 (1436149561585303552)]} 0 0
[junit4:junit4] 2> 248553 T1292 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[52 (1436149561585303552)]} 0 4
[junit4:junit4] 2> 248554 T1276 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[52]} 0 7
[junit4:junit4] 2> 248558 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[53 (1436149561593692160)]} 0 1
[junit4:junit4] 2> 248566 T1309 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[53 (1436149561599983616)]} 0 1
[junit4:junit4] 2> 248567 T1277 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[53 (1436149561599983616)]} 0 4
[junit4:junit4] 2> 248568 T1290 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[53]} 0 8
[junit4:junit4] 2> 248571 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[54 (1436149561608372224)]} 0 1
[junit4:junit4] 2> 248579 T1326 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[54 (1436149561613615104)]} 0 0
[junit4:junit4] 2> 248580 T1291 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[54 (1436149561613615104)]} 0 4
[junit4:junit4] 2> 248581 T1274 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[54]} 0 7
[junit4:junit4] 2> 248584 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[55 (1436149561622003712)]} 0 0
[junit4:junit4] 2> 248592 T1323 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[55 (1436149561627246592)]} 0 0
[junit4:junit4] 2> 248593 T1293 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[55 (1436149561627246592)]} 0 4
[junit4:junit4] 2> 248594 T1275 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[55]} 0 7
[junit4:junit4] 2> 248597 T1254 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[56 (1436149561635635200)]} 0 0
[junit4:junit4] 2> 248603 T1307 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[56 (1436149561638780928)]} 0 1
[junit4:junit4] 2> 248604 T1276 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[56 (1436149561638780928)]} 0 4
[junit4:junit4] 2> 248607 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[57 (1436149561646120960)]} 0 1
[junit4:junit4] 2> 248613 T1306 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[57 (1436149561649266688)]} 0 1
[junit4:junit4] 2> 248614 T1277 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[57 (1436149561649266688)]} 0 4
[junit4:junit4] 2> 248617 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[58 (1436149561656606720)]} 0 1
[junit4:junit4] 2> 248623 T1308 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[58 (1436149561659752448)]} 0 1
[junit4:junit4] 2> 248624 T1274 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[58 (1436149561659752448)]} 0 4
[junit4:junit4] 2> 248627 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[59 (1436149561667092480)]} 0 1
[junit4:junit4] 2> 248635 T1325 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[59 (1436149561672335360)]} 0 1
[junit4:junit4] 2> 248636 T1292 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[59 (1436149561672335360)]} 0 4
[junit4:junit4] 2> 248637 T1275 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[59]} 0 7
[junit4:junit4] 2> 248640 T1254 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[60 (1436149561680723968)]} 0 1
[junit4:junit4] 2> 248646 T1309 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[60 (1436149561683869696)]} 0 1
[junit4:junit4] 2> 248646 T1276 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[60 (1436149561683869696)]} 0 3
[junit4:junit4] 2> 248650 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[61 (1436149561690161152)]} 0 1
[junit4:junit4] 2> 248656 T1324 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[61 (1436149561694355456)]} 0 1
[junit4:junit4] 2> 248656 T1290 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[61 (1436149561694355456)]} 0 4
[junit4:junit4] 2> 248660 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[62 (1436149561700646912)]} 0 1
[junit4:junit4] 2> 248665 T1307 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[62 (1436149561704841216)]} 0 0
[junit4:junit4] 2> 248666 T1277 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[62 (1436149561704841216)]} 0 4
[junit4:junit4] 2> 248670 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[63 (1436149561711132672)]} 0 1
[junit4:junit4] 2> 248678 T1326 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[63 (1436149561717424128)]} 0 1
[junit4:junit4] 2> 248679 T1291 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[63 (1436149561717424128)]} 0 4
[junit4:junit4] 2> 248679 T1274 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[63]} 0 7
[junit4:junit4] 2> 248683 T1254 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[64 (1436149561724764160)]} 0 1
[junit4:junit4] 2> 248688 T1323 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[64 (1436149561727909888)]} 0 0
[junit4:junit4] 2> 248689 T1293 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[64 (1436149561727909888)]} 0 4
[junit4:junit4] 2> 248693 T1257 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[65 (1436149561735249920)]} 0 1
[junit4:junit4] 2> 248698 T1325 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[65 (1436149561738395648)]} 0 0
[junit4:junit4] 2> 248699 T1292 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[65 (1436149561738395648)]} 0 4
[junit4:junit4] 2> 248702 T1256 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[66 (1436149561745735680)]} 0 0
[junit4:junit4] 2> 248708 T1324 C327 P60747 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[66 (1436149561748881408)]} 0 0
[junit4:junit4] 2> 248709 T1290 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[66 (1436149561748881408)]} 0 4
[junit4:junit4] 2> 248712 T1255 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[67 (1436149561756221440)]} 0 0
[junit4:junit4] 2> 248721 T1306 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[67 (1436149561761464320)]} 0 0
[junit4:junit4] 2> 248721 T1275 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[67 (1436149561761464320)]} 0 4
[junit4:junit4] 2> 248722 T1293 C330 P51008 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[67]} 0 7
[junit4:junit4] 2> 248725 T1254 C331 P42207 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={wt=javabin&version=2} {add=[68 (1436149561769852928)]} 0 0
[junit4:junit4] 2> 248734 T1309 C328 P48513 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:50876/uv/b/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[68 (1436149561776144384)]} 0 1
[junit4:junit4] 2> 248735 T1276 C329 P50876 oasup.LogUpdateProcessor.finish [collection1] webapp=/uv/b path=/update params={distrib.from=http://127.0.0.1:51008/uv/b/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[68 (14361
[...truncated too long message...]
p://127.0.0.1:48513/uv/b",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "node_name":"127.0.0.1:48513_uv%2Fb"}
[junit4:junit4] 2> 382840 T1353 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:48513_uv%2Fb_collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48513/uv/b",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "node_name":"127.0.0.1:48513_uv%2Fb",
[junit4:junit4] 2> "shard_state":"active"}
[junit4:junit4] 2> 382847 T1239 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/uv/b,null}
[junit4:junit4] 2> 382850 T1330 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 382899 T1239 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 60747
[junit4:junit4] 2> 382899 T1239 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=8477731
[junit4:junit4] 2> 383040 T1281 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 383041 T1281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 383041 T1281 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 383901 T1239 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 383902 T1239 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 383903 T1239 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@10f6043
[junit4:junit4] 2> 383908 T1239 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=4,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=694,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 383908 T1239 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 383908 T1239 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 383909 T1239 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 383910 T1239 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 383911 T1239 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 383911 T1239 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty4/index;done=false>>]
[junit4:junit4] 2> 383911 T1239 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty4/index
[junit4:junit4] 2> 383912 T1239 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty4;done=false>>]
[junit4:junit4] 2> 383912 T1239 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882/jetty4
[junit4:junit4] 2> 383912 T1353 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89759345861328908-127.0.0.1:60747_uv%2Fb-n_0000000004) am no longer a leader.
[junit4:junit4] 2> 383934 T1239 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/uv/b,null}
[junit4:junit4] 2> 383990 T1239 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 383991 T1239 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:47398 47398
[junit4:junit4] 2> 383992 T1239 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 42207
[junit4:junit4] 2> !!!! WARNING: best effort to remove /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyShardSplitTest-1369618924882 FAILED !!!!!
[junit4:junit4] 2> 383993 T1239 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 383993 T1239 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:47398 47398
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeyShardSplitTest -Dtests.method=testDistribSearch -Dtests.seed=DC9515E394AF7C1C -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=it_IT -Dtests.timezone=Asia/Gaza -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE 162s J0 | ChaosMonkeyShardSplitTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard2 is not consistent. Got 49 from http://127.0.0.1:50876/uv/b/collection1lastClient and got 48 from http://127.0.0.1:48513/uv/b/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([DC9515E394AF7C1C:5D739BFBE3F01C20]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
[junit4:junit4] > at org.apache.solr.cloud.ChaosMonkeyShardSplitTest.doTest(ChaosMonkeyShardSplitTest.java:110)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 384030 T1239 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 162009 T1238 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
[junit4:junit4] 2> 384126 T1313 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 384127 T1313 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 384127 T1313 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 385213 T1330 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 385214 T1330 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 385214 T1330 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {_version_=Lucene41(blocksize=128), id=MockVariableIntBlock(baseBlockSize=114)}, docValues:{}, sim=DefaultSimilarity, locale=it_IT, timezone=Asia/Gaza
[junit4:junit4] 2> NOTE: Linux 3.2.0-41-generic i386/Oracle Corporation 1.8.0-ea (32-bit)/cpus=8,threads=1,free=151364880,total=233308160
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestDocumentBuilder, TestCharFilters, LoggingHandlerTest, BasicFunctionalityTest, TestFastWriter, TestSchemaVersionResource, TestLMDirichletSimilarityFactory, MBeansHandlerTest, SyncSliceTest, TestSerializedLuceneMatchVersion, DirectSolrConnectionTest, TestQueryUtils, AnalysisAfterCoreReloadTest, SpellingQueryConverterTest, BadComponentTest, TestBadConfig, DateFieldTest, SystemInfoHandlerTest, TestGroupingSearch, UUIDFieldTest, TestSolrQueryParserDefaultOperatorResource, CacheHeaderTest, SolrInfoMBeanTest, TestSolrIndexConfig, SliceStateUpdateTest, PrimitiveFieldTypeTest, TestXIncludeConfig, TestStressRecovery, DistributedQueryElevationComponentTest, RegexBoostProcessorTest, ResourceLoaderTest, TestFuzzyAnalyzedSuggestions, FieldMutatingUpdateProcessorTest, UpdateRequestProcessorFactoryTest, FastVectorHighlighterTest, TestReplicationHandler, DistributedSpellCheckComponentTest, SearchHandlerTest, SuggesterTest, TestValueSourceCache, TestQuerySenderListener, ChaosMonkeyShardSplitTest]
[junit4:junit4] Completed on J0 in 163.30s, 1 test, 1 failure <<< FAILURES!
[...truncated 553 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:380: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:360: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1240: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:884: There were test failures: 295 suites, 1231 tests, 1 failure, 16 ignored (7 assumptions)
Total time: 34 minutes 35 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-ea-b89 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_21) - Build #
5848 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/5848/
Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseParallelGC
1 tests failed.
FAILED: org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch
Error Message:
There are still nodes recoverying - waited for 230 seconds
Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 230 seconds
at __randomizedtesting.SeedInfo.seed([B87450A4B3E529F4:3992DEBCC4BA49C8]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:173)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:512)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:146)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9437 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
[junit4:junit4] 2> 79411 T314 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /r_em/ow
[junit4:junit4] 2> 79416 T314 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-CollectionsAPIDistributedZkTest-1369623826584
[junit4:junit4] 2> 79417 T314 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 79417 T315 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 79517 T314 oasc.ZkTestServer.run start zk server on port:33865
[junit4:junit4] 2> 79528 T314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 79621 T321 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71884b88 name:ZooKeeperConnection Watcher:127.0.0.1:33865 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 79622 T314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 79622 T314 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 79629 T314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 79630 T323 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66234339 name:ZooKeeperConnection Watcher:127.0.0.1:33865/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 79631 T314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 79631 T314 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 79634 T314 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 79638 T314 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 79640 T314 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 79643 T314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 79643 T314 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 79648 T314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 79648 T314 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 79652 T314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 79653 T314 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 79655 T314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 79656 T314 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 79659 T314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 79659 T314 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 79662 T314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 79662 T314 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 79664 T314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 79665 T314 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 79670 T314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 79671 T314 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 79674 T314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 79675 T314 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 79749 T314 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 79752 T314 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34139
[junit4:junit4] 2> 79753 T314 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 79753 T314 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 79753 T314 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846
[junit4:junit4] 2> 79754 T314 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/solr.xml
[junit4:junit4] 2> 79754 T314 oasc.CoreContainer.<init> New CoreContainer 1330877901
[junit4:junit4] 2> 79754 T314 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/'
[junit4:junit4] 2> 79754 T314 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/'
[junit4:junit4] 2> 79794 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846
[junit4:junit4] 2> 79795 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/collection1
[junit4:junit4] 2> 79795 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/collection1/conf
[junit4:junit4] 2> 79800 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/collection1/conf/xslt
[junit4:junit4] 2> 79816 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/collection1/lib
[junit4:junit4] 2> 79817 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/collection1/lib/classes
[junit4:junit4] 2> 79818 T314 oasc.SolrCoreDiscoverer.addCore Discovered properties file /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/conf/core.properties, adding to cores
[junit4:junit4] 2> 79820 T314 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4] 2> 79820 T314 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 79821 T314 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4] 2> 79821 T314 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 79821 T314 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 79822 T314 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 79822 T314 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 79822 T314 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 79823 T314 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 79823 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4] 2> 79827 T314 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 79827 T314 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33865/solr
[junit4:junit4] 2> 79828 T314 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 79829 T314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 79830 T334 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63787f67 name:ZooKeeperConnection Watcher:127.0.0.1:33865 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 79830 T314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 79832 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 79837 T314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 79844 T336 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4594c0ae name:ZooKeeperConnection Watcher:127.0.0.1:33865/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 79847 T314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 79849 T314 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 79852 T314 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 79854 T314 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 79856 T314 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34139_r_em%2Fow
[junit4:junit4] 2> 79857 T314 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34139_r_em%2Fow
[junit4:junit4] 2> 79861 T314 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 79866 T314 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 79868 T314 oasc.Overseer.start Overseer (id=89759667099140099-127.0.0.1:34139_r_em%2Fow-n_0000000000) starting
[junit4:junit4] 2> 79871 T314 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 79874 T338 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 79875 T314 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 79877 T314 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 79879 T314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 79883 T314 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 79884 T314 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 79884 T337 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 79903 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 79910 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 79913 T314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 79915 T341 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@499ec83d name:ZooKeeperConnection Watcher:127.0.0.1:33865/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 79915 T314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 79916 T314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 79920 T314 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 80002 T314 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 80004 T314 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53237
[junit4:junit4] 2> 80004 T314 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 80005 T314 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 80005 T314 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088
[junit4:junit4] 2> 80005 T314 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/solr.xml
[junit4:junit4] 2> 80006 T314 oasc.CoreContainer.<init> New CoreContainer 827250151
[junit4:junit4] 2> 80006 T314 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/'
[junit4:junit4] 2> 80007 T314 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/'
[junit4:junit4] 2> 80051 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088
[junit4:junit4] 2> 80052 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/collection1
[junit4:junit4] 2> 80052 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/collection1/conf
[junit4:junit4] 2> 80057 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/collection1/conf/xslt
[junit4:junit4] 2> 80081 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/collection1/lib
[junit4:junit4] 2> 80082 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/collection1/lib/classes
[junit4:junit4] 2> 80083 T314 oasc.SolrCoreDiscoverer.addCore Discovered properties file /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/conf/core.properties, adding to cores
[junit4:junit4] 2> 80086 T314 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4] 2> 80086 T314 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 80086 T314 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4] 2> 80087 T314 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 80087 T314 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 80087 T314 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 80088 T314 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 80088 T314 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 80088 T314 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 80089 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4] 2> 80093 T314 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 80093 T314 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33865/solr
[junit4:junit4] 2> 80093 T314 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 80094 T314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 80096 T352 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@698615e9 name:ZooKeeperConnection Watcher:127.0.0.1:33865 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 80097 T314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 80100 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 80106 T314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 80107 T354 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35397d84 name:ZooKeeperConnection Watcher:127.0.0.1:33865/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 80107 T314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 80111 T314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 81114 T314 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53237_r_em%2Fow
[junit4:junit4] 2> 81116 T314 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53237_r_em%2Fow
[junit4:junit4] 2> 81121 T341 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 81122 T354 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 81122 T336 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 81131 T314 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 81131 T314 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 81132 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 81204 T314 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 81206 T314 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37449
[junit4:junit4] 2> 81206 T314 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 81207 T314 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 81207 T314 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302
[junit4:junit4] 2> 81207 T314 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/solr.xml
[junit4:junit4] 2> 81207 T314 oasc.CoreContainer.<init> New CoreContainer 589272953
[junit4:junit4] 2> 81208 T314 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/'
[junit4:junit4] 2> 81208 T314 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/'
[junit4:junit4] 2> 81243 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302
[junit4:junit4] 2> 81244 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/collection1
[junit4:junit4] 2> 81245 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/collection1/conf
[junit4:junit4] 2> 81249 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/collection1/conf/xslt
[junit4:junit4] 2> 81263 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/collection1/lib
[junit4:junit4] 2> 81264 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/collection1/lib/classes
[junit4:junit4] 2> 81265 T314 oasc.SolrCoreDiscoverer.addCore Discovered properties file /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/conf/core.properties, adding to cores
[junit4:junit4] 2> 81268 T314 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4] 2> 81268 T314 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 81268 T314 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4] 2> 81268 T314 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 81269 T314 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 81269 T314 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 81269 T314 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 81270 T314 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 81270 T314 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 81270 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4] 2> 81274 T314 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 81274 T314 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33865/solr
[junit4:junit4] 2> 81275 T314 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 81275 T314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 81277 T366 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69e2c341 name:ZooKeeperConnection Watcher:127.0.0.1:33865 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 81277 T314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 81279 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 81285 T314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 81287 T368 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2bfa9322 name:ZooKeeperConnection Watcher:127.0.0.1:33865/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 81287 T314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 81292 T314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 82296 T314 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37449_r_em%2Fow
[junit4:junit4] 2> 82299 T314 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37449_r_em%2Fow
[junit4:junit4] 2> 82301 T354 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 82302 T368 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 82302 T341 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 82302 T336 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 82306 T314 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 82306 T314 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 82307 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 82395 T314 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 82398 T314 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52867
[junit4:junit4] 2> 82398 T314 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 82399 T314 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 82399 T314 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477
[junit4:junit4] 2> 82399 T314 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/solr.xml
[junit4:junit4] 2> 82399 T314 oasc.CoreContainer.<init> New CoreContainer 140433380
[junit4:junit4] 2> 82400 T314 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/'
[junit4:junit4] 2> 82400 T314 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/'
[junit4:junit4] 2> 82436 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477
[junit4:junit4] 2> 82437 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/collection1
[junit4:junit4] 2> 82437 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/collection1/conf
[junit4:junit4] 2> 82441 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/collection1/conf/xslt
[junit4:junit4] 2> 82456 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/collection1/lib
[junit4:junit4] 2> 82456 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/collection1/lib/classes
[junit4:junit4] 2> 82458 T314 oasc.SolrCoreDiscoverer.addCore Discovered properties file /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/conf/core.properties, adding to cores
[junit4:junit4] 2> 82459 T314 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4] 2> 82460 T314 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 82460 T314 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4] 2> 82460 T314 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 82461 T314 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 82461 T314 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 82461 T314 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 82461 T314 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 82462 T314 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 82462 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4] 2> 82466 T314 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 82466 T314 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33865/solr
[junit4:junit4] 2> 82466 T314 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 82467 T314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 82468 T380 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a3ccb09 name:ZooKeeperConnection Watcher:127.0.0.1:33865 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 82469 T314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 82470 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 82474 T314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 82476 T382 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ff66404 name:ZooKeeperConnection Watcher:127.0.0.1:33865/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 82476 T314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 82479 T314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 83483 T314 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52867_r_em%2Fow
[junit4:junit4] 2> 83485 T314 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52867_r_em%2Fow
[junit4:junit4] 2> 83487 T354 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 83488 T336 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 83488 T341 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 83488 T368 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 83488 T382 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 83492 T314 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 83493 T314 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 83493 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 83570 T314 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 83572 T314 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38602
[junit4:junit4] 2> 83573 T314 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 83573 T314 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 83574 T314 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663
[junit4:junit4] 2> 83574 T314 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/solr.xml
[junit4:junit4] 2> 83574 T314 oasc.CoreContainer.<init> New CoreContainer 250532119
[junit4:junit4] 2> 83575 T314 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/'
[junit4:junit4] 2> 83575 T314 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/'
[junit4:junit4] 2> 83619 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663
[junit4:junit4] 2> 83620 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/collection1
[junit4:junit4] 2> 83620 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/collection1/conf
[junit4:junit4] 2> 83625 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/collection1/conf/xslt
[junit4:junit4] 2> 83640 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/collection1/lib
[junit4:junit4] 2> 83641 T314 oasc.SolrCoreDiscoverer.walkFromHere Looking for cores in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/collection1/lib/classes
[junit4:junit4] 2> 83642 T314 oasc.SolrCoreDiscoverer.addCore Discovered properties file /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/conf/core.properties, adding to cores
[junit4:junit4] 2> 83644 T314 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4] 2> 83644 T314 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 83644 T314 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4] 2> 83645 T314 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 83645 T314 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 83645 T314 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 83646 T314 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 83646 T314 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 83646 T314 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 83647 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4] 2> 83650 T314 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 83651 T314 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33865/solr
[junit4:junit4] 2> 83651 T314 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 83652 T314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 83653 T394 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26be54c8 name:ZooKeeperConnection Watcher:127.0.0.1:33865 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 83654 T314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 83655 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 83659 T314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 83660 T396 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b5527e5 name:ZooKeeperConnection Watcher:127.0.0.1:33865/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 83660 T314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 83664 T314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 84667 T314 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38602_r_em%2Fow
[junit4:junit4] 2> 84669 T314 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38602_r_em%2Fow
[junit4:junit4] 2> 84672 T382 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 84672 T341 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 84673 T368 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 84673 T336 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 84674 T396 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 84674 T354 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 84683 T314 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 84683 T314 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 84685 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 84695 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 84917 T349 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&name=nodes_used_collection&replicationFactor=2&action=CREATE&wt=javabin&version=2
[junit4:junit4] 2> 84919 T336 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4] 2> 84922 T338 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
[junit4:junit4] 2> "operation":"createcollection",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "maxShardsPerNode":null,
[junit4:junit4] 2> "createNodeSet":null,
[junit4:junit4] 2> "name":"nodes_used_collection",
[junit4:junit4] 2> "replicationFactor":"2"}
[junit4:junit4] 2> 84923 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica1 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:52867_r_em%2Fow
[junit4:junit4] 2> 84927 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica2 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:37449_r_em%2Fow
[junit4:junit4] 2> 84928 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica1 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:53237_r_em%2Fow
[junit4:junit4] 2> 84929 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica2 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:34139_r_em%2Fow
[junit4:junit4] 2> 84940 T347 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/nodes_used_collection_shard2_replica1
[junit4:junit4] 2> 84941 T362 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/nodes_used_collection_shard1_replica2
[junit4:junit4] 2> 84944 T362 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4] 2> 84940 T376 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/nodes_used_collection_shard1_replica1
[junit4:junit4] 2> 84946 T376 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4] 2> 84941 T332 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/nodes_used_collection_shard2_replica2
[junit4:junit4] 2> 84949 T332 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4] 2> 84948 T376 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4] 2> 84946 T362 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4] 2> 84942 T347 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
[junit4:junit4] 2> 84952 T362 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 84951 T376 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 84950 T332 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4] 2> 84956 T376 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 84955 T362 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 84954 T347 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
[junit4:junit4] 2> 84958 T362 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4] 2> 84958 T376 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4] 2> 84957 T332 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 84959 T347 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 84962 T376 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4] 2> 84962 T347 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4] 2> 84962 T332 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
[junit4:junit4] 2> 84962 T362 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4] 2> 84965 T347 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4] 2> 84964 T376 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/nodes_used_collection_shard1_replica1/'
[junit4:junit4] 2> 84967 T362 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/nodes_used_collection_shard1_replica2/'
[junit4:junit4] 2> 84967 T347 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/nodes_used_collection_shard2_replica1/'
[junit4:junit4] 2> 84966 T332 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
[junit4:junit4] 2> 84971 T332 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/nodes_used_collection_shard2_replica2/'
[junit4:junit4] 2> 85017 T362 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 85018 T376 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 85021 T347 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 85026 T332 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 85082 T362 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 85084 T347 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 85085 T362 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 85086 T376 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 85086 T347 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 85088 T376 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 85090 T332 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 85090 T362 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica2] Schema name=test
[junit4:junit4] 2> 85092 T347 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica1] Schema name=test
[junit4:junit4] 2> 85093 T332 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 85094 T376 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica1] Schema name=test
[junit4:junit4] 2> 85098 T332 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica2] Schema name=test
[junit4:junit4] 2> 85452 T347 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 85459 T347 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 85462 T347 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 85475 T347 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 85480 T347 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 85485 T347 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 85486 T347 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 85487 T347 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 85487 T347 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 85489 T347 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 85489 T347 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 85490 T347 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 85490 T347 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/nodes_used_collection_shard2_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/nodes_used_collection_shard2_replica1/data/
[junit4:junit4] 2> 85491 T347 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@14daf5d0
[junit4:junit4] 2> 85493 T347 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 85494 T347 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/nodes_used_collection_shard2_replica1/data
[junit4:junit4] 2> 85494 T347 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/nodes_used_collection_shard2_replica1/data/index/
[junit4:junit4] 2> 85494 T347 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/nodes_used_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 85495 T347 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/nodes_used_collection_shard2_replica1/data/index
[junit4:junit4] 2> 85497 T347 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@41e0aacd lockFactory=org.apache.lucene.store.NativeFSLockFactory@74848c08),segFN=segments_1,generation=1}
[junit4:junit4] 2> 85497 T347 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 85500 T347 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 85500 T347 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 85501 T347 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 85502 T347 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 85503 T347 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 85503 T347 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 85504 T347 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 85504 T347 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 85505 T347 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 85508 T347 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 85511 T347 oass.SolrIndexSearcher.<init> Opening Searcher@5889c5eb main
[junit4:junit4] 2> 85512 T347 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/nodes_used_collection_shard2_replica1/data/tlog
[junit4:junit4] 2> 85513 T347 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 85513 T347 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 85518 T402 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica1] Registered new searcher Searcher@5889c5eb main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 85520 T347 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica1 state=down
[junit4:junit4] 2> 85564 T332 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 85565 T362 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 85570 T362 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 85572 T332 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 85573 T362 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 85576 T332 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 85587 T376 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 85588 T362 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 85591 T332 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 85593 T362 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 85593 T376 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 85596 T376 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 85596 T332 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 85596 T362 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 85598 T362 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 85600 T362 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 85600 T362 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 85602 T362 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 85602 T362 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 85602 T362 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 85603 T362 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/nodes_used_collection_shard1_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/nodes_used_collection_shard1_replica2/data/
[junit4:junit4] 2> 85603 T332 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 85603 T362 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@14daf5d0
[junit4:junit4] 2> 85604 T362 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 85605 T332 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 85605 T362 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/nodes_used_collection_shard1_replica2/data
[junit4:junit4] 2> 85605 T332 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 85605 T362 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/nodes_used_collection_shard1_replica2/data/index/
[junit4:junit4] 2> 85606 T332 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 85606 T362 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/nodes_used_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 85607 T362 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/nodes_used_collection_shard1_replica2/data/index
[junit4:junit4] 2> 85607 T332 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 85608 T332 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 85608 T332 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 85608 T362 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@b671b66 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4f5672d1),segFN=segments_1,generation=1}
[junit4:junit4] 2> 85609 T362 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 85608 T332 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/nodes_used_collection_shard2_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/nodes_used_collection_shard2_replica2/data/
[junit4:junit4] 2> 85609 T332 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@14daf5d0
[junit4:junit4] 2> 85610 T332 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 85610 T376 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 85611 T362 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 85611 T332 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/nodes_used_collection_shard2_replica2/data
[junit4:junit4] 2> 85611 T362 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 85611 T332 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/nodes_used_collection_shard2_replica2/data/index/
[junit4:junit4] 2> 85612 T362 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 85612 T332 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/nodes_used_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 85612 T362 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 85613 T332 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/nodes_used_collection_shard2_replica2/data/index
[junit4:junit4] 2> 85613 T362 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 85613 T362 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 85613 T362 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 85614 T362 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 85614 T362 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 85615 T332 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7933c3af lockFactory=org.apache.lucene.store.NativeFSLockFactory@620831c7),segFN=segments_1,generation=1}
[junit4:junit4] 2> 85615 T376 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 85615 T332 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 85616 T362 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 85618 T332 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 85618 T332 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 85619 T362 oass.SolrIndexSearcher.<init> Opening Searcher@184004b5 main
[junit4:junit4] 2> 85619 T362 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/nodes_used_collection_shard1_replica2/data/tlog
[junit4:junit4] 2> 85619 T332 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 85619 T362 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 85619 T376 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 85620 T332 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 85620 T362 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 85621 T332 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 85621 T332 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 85621 T376 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 85621 T376 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 85621 T332 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 85622 T376 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 85622 T332 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 85623 T332 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 85623 T376 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 85623 T376 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 85623 T403 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica2] Registered new searcher Searcher@184004b5 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 85624 T376 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 85624 T376 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/nodes_used_collection_shard1_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/nodes_used_collection_shard1_replica1/data/
[junit4:junit4] 2> 85624 T362 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica2 state=down
[junit4:junit4] 2> 85624 T376 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@14daf5d0
[junit4:junit4] 2> 85625 T376 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 85626 T332 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 85626 T376 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/nodes_used_collection_shard1_replica1/data
[junit4:junit4] 2> 85626 T376 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/nodes_used_collection_shard1_replica1/data/index/
[junit4:junit4] 2> 85626 T376 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/nodes_used_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 85627 T376 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/nodes_used_collection_shard1_replica1/data/index
[junit4:junit4] 2> 85628 T332 oass.SolrIndexSearcher.<init> Opening Searcher@579eaf6a main
[junit4:junit4] 2> 85628 T376 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@66568940 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4da5181a),segFN=segments_1,generation=1}
[junit4:junit4] 2> 85629 T376 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 85629 T332 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/nodes_used_collection_shard2_replica2/data/tlog
[junit4:junit4] 2> 85630 T332 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 85630 T332 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 85630 T376 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 85631 T376 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 85631 T376 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 85632 T376 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 85632 T376 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 85632 T376 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 85633 T376 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 85633 T376 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 85633 T404 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica2] Registered new searcher Searcher@579eaf6a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 85633 T376 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 85634 T332 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica2 state=down
[junit4:junit4] 2> 85636 T376 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 85638 T376 oass.SolrIndexSearcher.<init> Opening Searcher@333d3c5d main
[junit4:junit4] 2> 85639 T376 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/nodes_used_collection_shard1_replica1/data/tlog
[junit4:junit4] 2> 85639 T376 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 85639 T376 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 85642 T405 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica1] Registered new searcher Searcher@333d3c5d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 85643 T376 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica1 state=down
[junit4:junit4] 2> 85896 T337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 85897 T337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"nodes_used_collection_shard2_replica1",
[junit4:junit4] 2> "collection":"nodes_used_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53237_r_em%2Fow",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53237/r_em/ow"}
[junit4:junit4] 2> 85897 T337 oasc.Overseer$ClusterStateUpdater.createCollection Create collection nodes_used_collection with numShards 2
[junit4:junit4] 2> 85902 T337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"nodes_used_collection_shard1_replica2",
[junit4:junit4] 2> "collection":"nodes_used_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:37449_r_em%2Fow",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:37449/r_em/ow"}
[junit4:junit4] 2> 85906 T337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"nodes_used_collection_shard2_replica2",
[junit4:junit4] 2> "collection":"nodes_used_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:34139_r_em%2Fow",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:34139/r_em/ow"}
[junit4:junit4] 2> 85909 T337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"nodes_used_collection_shard1_replica1",
[junit4:junit4] 2> "collection":"nodes_used_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52867_r_em%2Fow",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52867/r_em/ow"}
[junit4:junit4] 2> 85914 T382 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 85914 T354 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 85914 T368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 85914 T336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 85914 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 85915 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 86551 T347 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica1
[junit4:junit4] 2> 86551 T347 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica1 address:http://127.0.0.1:53237/r_em/ow collection:nodes_used_collection shard:shard2
[junit4:junit4] 2> 86554 T347 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 86559 T347 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard2/election
[junit4:junit4] 2> 86565 T347 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 86568 T347 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up: total=2 found=1 timeoutin=179999
[junit4:junit4] 2> 86627 T362 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard1_replica2
[junit4:junit4] 2> 86627 T362 oasc.ZkController.register Register replica - core:nodes_used_collection_shard1_replica2 address:http://127.0.0.1:37449/r_em/ow collection:nodes_used_collection shard:shard1
[junit4:junit4] 2> 86628 T362 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard1/election
[junit4:junit4] 2> 86635 T362 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 86636 T332 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica2
[junit4:junit4] 2> 86636 T332 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica2 address:http://127.0.0.1:34139/r_em/ow collection:nodes_used_collection shard:shard2
[junit4:junit4] 2> 86638 T362 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up: total=2 found=1 timeoutin=179999
[junit4:junit4] 2> 86644 T376 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard1_replica1
[junit4:junit4] 2> 86645 T376 oasc.ZkController.register Register replica - core:nodes_used_collection_shard1_replica1 address:http://127.0.0.1:52867/r_em/ow collection:nodes_used_collection shard:shard1
[junit4:junit4] 2> 87069 T347 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 87069 T347 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 87071 T347 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53237/r_em/ow/nodes_used_collection_shard2_replica1/
[junit4:junit4] 2> 87074 T347 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 87078 T347 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard2_replica1 url=http://127.0.0.1:53237/r_em/ow START replicas=[http://127.0.0.1:34139/r_em/ow/nodes_used_collection_shard2_replica2/] nUpdates=100
[junit4:junit4] 2> 87079 T347 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard2_replica1 url=http://127.0.0.1:53237/r_em/ow DONE. We have no versions. sync failed.
[junit4:junit4] 2> 87079 T347 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next canidate
[junit4:junit4] 2> 87080 T347 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4:junit4] 2> 87080 T347 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53237/r_em/ow/nodes_used_collection_shard2_replica1/
[junit4:junit4] 2> 87080 T347 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leaders/shard2
[junit4:junit4] 2> ASYNC NEW_CORE C80 name=nodes_used_collection_shard2_replica2 org.apache.solr.core.SolrCore@3db90619 url=http://127.0.0.1:34139/r_em/ow/nodes_used_collection_shard2_replica2 node=127.0.0.1:34139_r_em%2Fow C80_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica2 props:{shard=shard2, state=down, core=nodes_used_collection_shard2_replica2, collection=nodes_used_collection, node_name=127.0.0.1:34139_r_em%2Fow, base_url=http://127.0.0.1:34139/r_em/ow}
[junit4:junit4] 2> 87088 T330 C80 P34139 oasc.SolrCore.execute [nodes_used_collection_shard2_replica2] webapp=/r_em/ow path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=3
[junit4:junit4] 2> 87139 T362 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 87139 T362 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 87139 T362 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:37449/r_em/ow/nodes_used_collection_shard1_replica2/
[junit4:junit4] 2> 87140 T362 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard1_replica2 url=http://127.0.0.1:37449/r_em/ow START replicas=[http://127.0.0.1:52867/r_em/ow/nodes_used_collection_shard1_replica1/] nUpdates=100
[junit4:junit4] 2> 87141 T362 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard1_replica2 url=http://127.0.0.1:37449/r_em/ow DONE. We have no versions. sync failed.
[junit4:junit4] 2> 87141 T362 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next canidate
[junit4:junit4] 2> 87141 T362 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4:junit4] 2> 87141 T362 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:37449/r_em/ow/nodes_used_collection_shard1_replica2/
[junit4:junit4] 2> 87142 T362 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leaders/shard1
[junit4:junit4] 2> ASYNC NEW_CORE C81 name=nodes_used_collection_shard1_replica1 org.apache.solr.core.SolrCore@2b8d94aa url=http://127.0.0.1:52867/r_em/ow/nodes_used_collection_shard1_replica1 node=127.0.0.1:52867_r_em%2Fow C81_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica1 props:{shard=shard1, state=down, core=nodes_used_collection_shard1_replica1, collection=nodes_used_collection, node_name=127.0.0.1:52867_r_em%2Fow, base_url=http://127.0.0.1:52867/r_em/ow}
[junit4:junit4] 2> 87146 T377 C81 P52867 oasc.SolrCore.execute [nodes_used_collection_shard1_replica1] webapp=/r_em/ow path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 87424 T337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 87440 T354 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 87441 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 87441 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 87440 T382 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 87441 T336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 87441 T368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 87449 T362 oasc.ZkController.register We are http://127.0.0.1:37449/r_em/ow/nodes_used_collection_shard1_replica2/ and leader is http://127.0.0.1:37449/r_em/ow/nodes_used_collection_shard1_replica2/
[junit4:junit4] 2> 87450 T362 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard1_replica2 baseURL=http://127.0.0.1:37449/r_em/ow
[junit4:junit4] 2> 87450 T362 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 87450 T362 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica2 state=active
[junit4:junit4] 2> 87452 T362 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 87453 T362 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard1&name=nodes_used_collection_shard1_replica2&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=2513
[junit4:junit4] 2> 87488 T347 oasc.ZkController.register We are http://127.0.0.1:53237/r_em/ow/nodes_used_collection_shard2_replica1/ and leader is http://127.0.0.1:53237/r_em/ow/nodes_used_collection_shard2_replica1/
[junit4:junit4] 2> 87489 T347 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard2_replica1 baseURL=http://127.0.0.1:53237/r_em/ow
[junit4:junit4] 2> 87489 T347 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 87489 T347 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica1 state=active
[junit4:junit4] 2> 87491 T347 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 87492 T347 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard2&name=nodes_used_collection_shard2_replica1&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=2552
[junit4:junit4] 2> 87641 T332 oasc.ZkController.register We are http://127.0.0.1:34139/r_em/ow/nodes_used_collection_shard2_replica2/ and leader is http://127.0.0.1:53237/r_em/ow/nodes_used_collection_shard2_replica1/
[junit4:junit4] 2> 87642 T332 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard2_replica2 baseURL=http://127.0.0.1:34139/r_em/ow
[junit4:junit4] 2> 87642 T332 oasc.ZkController.checkRecovery Core needs to recover:nodes_used_collection_shard2_replica2
[junit4:junit4] 2> 87642 T332 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> 87646 T408 C80 P34139 oasc.RecoveryStrategy.run Starting recovery process. core=nodes_used_collection_shard2_replica2 recoveringAfterStartup=true
[junit4:junit4] 2> 87646 T408 C80 P34139 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 87647 T332 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 87647 T408 C80 P34139 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica2 state=recovering
[junit4:junit4] 2> 87648 T332 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard2&name=nodes_used_collection_shard2_replica2&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=2708
[junit4:junit4] 2> 87649 T408 C80 P34139 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 87649 T376 oasc.ZkController.register We are http://127.0.0.1:52867/r_em/ow/nodes_used_collection_shard1_replica1/ and leader is http://127.0.0.1:37449/r_em/ow/nodes_used_collection_shard1_replica2/
[junit4:junit4] 2> 87650 T376 oasc.ZkController.register No LogReplay needed for core=nodes_used_collection_shard1_replica1 baseURL=http://127.0.0.1:52867/r_em/ow
[junit4:junit4] 2> 87650 T376 oasc.ZkController.checkRecovery Core needs to recover:nodes_used_collection_shard1_replica1
[junit4:junit4] 2> 87650 T376 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> 87651 T409 C81 P52867 oasc.RecoveryStrategy.run Starting recovery process. core=nodes_used_collection_shard1_replica1 recoveringAfterStartup=true
[junit4:junit4] 2> 87652 T409 C81 P52867 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 87652 T409 C81 P52867 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica1 state=recovering
[junit4:junit4] 2> 87653 T376 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 87654 T409 C81 P52867 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 87654 T376 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=2&shard=shard1&name=nodes_used_collection_shard1_replica1&action=CREATE&collection=nodes_used_collection&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=2714
[junit4:junit4] 2> 87656 T338 oasc.OverseerCollectionProcessor.createCollection Finished create command on all shards for collection: nodes_used_collection
[junit4:junit4] 2> 87660 T348 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:34139_r_em%2Fow_nodes_used_collection_shard2_replica2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 87662 T363 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:52867_r_em%2Fow_nodes_used_collection_shard1_replica1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 87663 T354 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work/qnr-0000000000 state: SyncConnected type NodeDataChanged
[junit4:junit4] 2> 87665 T338 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Message id:/overseer/collection-queue-work/qn-0000000000 complete, response:{success={null={responseHeader={status=0,QTime=2513},core=nodes_used_collection_shard1_replica2},null={responseHeader={status=0,QTime=2552},core=nodes_used_collection_shard2_replica1},null={responseHeader={status=0,QTime=2708},core=nodes_used_collection_shard2_replica2},null={responseHeader={status=0,QTime=2714},core=nodes_used_collection_shard1_replica1}}}
[junit4:junit4] 2> 87665 T336 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4] 2> 87668 T349 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={numShards=2&name=nodes_used_collection&replicationFactor=2&action=CREATE&wt=javabin&version=2} status=0 QTime=2753
[junit4:junit4] 2> 87669 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 87672 T314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 87672 T411 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c16952a name:ZooKeeperConnection Watcher:127.0.0.1:33865/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 87672 T314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 87674 T314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 87677 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 87678 T314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 87679 T413 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76ac287b name:ZooKeeperConnection Watcher:127.0.0.1:33865/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 87680 T314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 87681 T314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 87690 T347 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=1&maxShardsPerNode=2&name=awholynewcollection_0&replicationFactor=5&action=CREATE&wt=javabin&version=2
[junit4:junit4] 2> 87691 T336 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4] 2> 87693 T338 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000002 message:{
[junit4:junit4] 2> "operation":"createcollection",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "maxShardsPerNode":"2",
[junit4:junit4] 2> "createNodeSet":null,
[junit4:junit4] 2> "name":"awholynewcollection_0",
[junit4:junit4] 2> "replicationFactor":"5"}
[junit4:junit4] 2> 87693 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica1 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:37449_r_em%2Fow
[junit4:junit4] 2> 87693 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica2 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:53237_r_em%2Fow
[junit4:junit4] 2> 87693 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica3 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:38602_r_em%2Fow
[junit4:junit4] 2> 87693 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica4 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:34139_r_em%2Fow
[junit4:junit4] 2> 87694 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_0_shard1_replica5 as part of slice shard1 of collection awholynewcollection_0 on 127.0.0.1:52867_r_em%2Fow
[junit4:junit4] 2> 87696 T392 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica3' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/awholynewcollection_0_shard1_replica3
[junit4:junit4] 2> 87696 T392 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4] 2> 87697 T350 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/awholynewcollection_0_shard1_replica2
[junit4:junit4] 2> 87697 T350 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4] 2> 87697 T377 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica5' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/awholynewcollection_0_shard1_replica5
[junit4:junit4] 2> 87698 T330 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica4' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/awholynewcollection_0_shard1_replica4
[junit4:junit4] 2> 87698 T330 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4] 2> 87698 T392 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4] 2> 87698 T377 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4] 2> 87698 T362 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_0_shard1_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/awholynewcollection_0_shard1_replica1
[junit4:junit4] 2> 87699 T362 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_0
[junit4:junit4] 2> 87699 T330 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4] 2> 87699 T392 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 87698 T350 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4] 2> 87700 T350 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 87700 T362 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4] 2> 87700 T330 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 87699 T377 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_0
[junit4:junit4] 2> 87701 T362 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 87701 T392 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 87702 T350 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 87702 T392 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4] 2> 87701 T377 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 87702 T350 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4] 2> 87702 T330 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 87704 T330 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4] 2> 87703 T362 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 87704 T377 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 87705 T377 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4] 2> 87704 T362 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0
[junit4:junit4] 2> 87706 T350 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4] 2> 87706 T392 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4] 2> 87707 T330 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4] 2> 87708 T362 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4] 2> 87708 T350 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/awholynewcollection_0_shard1_replica2/'
[junit4:junit4] 2> 87713 T377 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_0
[junit4:junit4] 2> 87713 T392 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/awholynewcollection_0_shard1_replica3/'
[junit4:junit4] 2> 87713 T330 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/awholynewcollection_0_shard1_replica4/'
[junit4:junit4] 2> 87714 T362 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/awholynewcollection_0_shard1_replica1/'
[junit4:junit4] 2> 87714 T377 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/awholynewcollection_0_shard1_replica5/'
[junit4:junit4] 2> 87752 T350 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 87754 T392 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 87756 T330 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 87758 T377 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 87777 T362 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 87782 T350 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 87784 T350 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 87793 T350 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica2] Schema name=test
[junit4:junit4] 2> 87802 T392 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 87804 T392 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 87804 T330 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 87806 T330 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 87808 T377 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 87809 T392 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica3] Schema name=test
[junit4:junit4] 2> 87810 T330 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica4] Schema name=test
[junit4:junit4] 2> 87810 T377 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 87815 T377 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica5] Schema name=test
[junit4:junit4] 2> 87844 T362 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 87859 T362 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 87865 T362 oass.IndexSchema.readSchema [awholynewcollection_0_shard1_replica1] Schema name=test
[junit4:junit4] 2> 88289 T350 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 88298 T350 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 88301 T392 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 88302 T350 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 88307 T377 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 88310 T392 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 88318 T377 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 88318 T392 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 88320 T377 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 88326 T330 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 88332 T330 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 88333 T362 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 88335 T330 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 88335 T350 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 88341 T362 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 88342 T350 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 88344 T362 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 88347 T392 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 88347 T350 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 88347 T377 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 88348 T350 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 88349 T350 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 88349 T350 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 88350 T350 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 88351 T350 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 88351 T392 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 88352 T377 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 88351 T350 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 88352 T330 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 88353 T350 oasc.SolrCore.<init> [awholynewcollection_0_shard1_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/awholynewcollection_0_shard1_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/awholynewcollection_0_shard1_replica2/data/
[junit4:junit4] 2> 88353 T350 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@14daf5d0
[junit4:junit4] 2> 88354 T350 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 88355 T350 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/awholynewcollection_0_shard1_replica2/data
[junit4:junit4] 2> 88355 T350 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/awholynewcollection_0_shard1_replica2/data/index/
[junit4:junit4] 2> 88355 T350 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard1_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/awholynewcollection_0_shard1_replica2/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 88356 T392 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 88356 T377 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 88356 T350 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/awholynewcollection_0_shard1_replica2/data/index
[junit4:junit4] 2> 88357 T392 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 88358 T392 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 88357 T377 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 88358 T377 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 88357 T330 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 88358 T377 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 88358 T392 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 88359 T350 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@43afbace lockFactory=org.apache.lucene.store.NativeFSLockFactory@6ddead5c),segFN=segments_1,generation=1}
[junit4:junit4] 2> 88360 T350 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 88360 T362 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 88360 T377 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 88361 T377 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 88361 T392 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 88366 T392 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 88366 T377 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 88363 T362 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 88363 T330 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 88362 T350 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 88367 T377 oasc.SolrCore.<init> [awholynewcollection_0_shard1_replica5] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/awholynewcollection_0_shard1_replica5/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/awholynewcollection_0_shard1_replica5/data/
[junit4:junit4] 2> 88367 T392 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 88368 T377 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@14daf5d0
[junit4:junit4] 2> 88368 T350 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 88369 T377 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 88368 T330 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 88369 T330 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 88368 T392 oasc.SolrCore.<init> [awholynewcollection_0_shard1_replica3] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/awholynewcollection_0_shard1_replica3/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/awholynewcollection_0_shard1_replica3/data/
[junit4:junit4] 2> 88370 T392 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@14daf5d0
[junit4:junit4] 2> 88369 T330 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 88369 T377 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/awholynewcollection_0_shard1_replica5/data
[junit4:junit4] 2> 88369 T350 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 88371 T377 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/awholynewcollection_0_shard1_replica5/data/index/
[junit4:junit4] 2> 88371 T350 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 88370 T392 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 88372 T350 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 88372 T350 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 88370 T362 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 88372 T350 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 88372 T392 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/awholynewcollection_0_shard1_replica3/data
[junit4:junit4] 2> 88373 T392 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/awholynewcollection_0_shard1_replica3/data/index/
[junit4:junit4] 2> 88372 T330 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 88374 T330 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 88371 T377 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard1_replica5] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/awholynewcollection_0_shard1_replica5/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 88374 T330 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 88374 T362 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 88375 T362 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 88374 T392 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard1_replica3] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/awholynewcollection_0_shard1_replica3/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 88373 T350 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 88376 T392 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/awholynewcollection_0_shard1_replica3/data/index
[junit4:junit4] 2> 88376 T362 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 88375 T330 oasc.SolrCore.<init> [awholynewcollection_0_shard1_replica4] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/awholynewcollection_0_shard1_replica4/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/awholynewcollection_0_shard1_replica4/data/
[junit4:junit4] 2> 88377 T330 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@14daf5d0
[junit4:junit4] 2> 88375 T377 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/awholynewcollection_0_shard1_replica5/data/index
[junit4:junit4] 2> 88378 T362 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 88378 T362 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 88377 T330 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 88376 T350 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 88378 T362 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 88378 T392 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@443a8adb lockFactory=org.apache.lucene.store.NativeFSLockFactory@4105a79c),segFN=segments_1,generation=1}
[junit4:junit4] 2> 88379 T392 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 88379 T362 oasc.SolrCore.<init> [awholynewcollection_0_shard1_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/awholynewcollection_0_shard1_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/awholynewcollection_0_shard1_replica1/data/
[junit4:junit4] 2> 88379 T377 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@78546d0c lockFactory=org.apache.lucene.store.NativeFSLockFactory@7547c27),segFN=segments_1,generation=1}
[junit4:junit4] 2> 88380 T377 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 88379 T330 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/awholynewcollection_0_shard1_replica4/data
[junit4:junit4] 2> 88381 T330 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/awholynewcollection_0_shard1_replica4/data/index/
[junit4:junit4] 2> 88381 T330 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard1_replica4] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/awholynewcollection_0_shard1_replica4/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 88381 T350 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 88380 T362 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@14daf5d0
[junit4:junit4] 2> 88381 T330 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/awholynewcollection_0_shard1_replica4/data/index
[junit4:junit4] 2> 88381 T392 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 88382 T377 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 88382 T362 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 88383 T377 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 88382 T392 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 88384 T330 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4f898cfc lockFactory=org.apache.lucene.store.NativeFSLockFactory@5a8f1d68),segFN=segments_1,generation=1}
[junit4:junit4] 2> 88384 T392 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 88383 T350 oass.SolrIndexSearcher.<init> Opening Searcher@7d9b17f7 main
[junit4:junit4] 2> 88383 T362 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/awholynewcollection_0_shard1_replica1/data
[junit4:junit4] 2> 88385 T362 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/awholynewcollection_0_shard1_replica1/data/index/
[junit4:junit4] 2> 88385 T362 oasc.SolrCore.initIndex WARN [awholynewcollection_0_shard1_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/awholynewcollection_0_shard1_replica1/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 88385 T350 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/awholynewcollection_0_shard1_replica2/data/tlog
[junit4:junit4] 2> 88384 T392 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 88384 T330 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 88384 T377 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 88386 T392 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 88387 T392 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 88386 T350 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 88385 T362 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/awholynewcollection_0_shard1_replica1/data/index
[junit4:junit4] 2> 88387 T350 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 88387 T392 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 88387 T377 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 88388 T392 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 88389 T377 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 88389 T377 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 88388 T330 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 88389 T377 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 88389 T392 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 88389 T362 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4d0ad3b2 lockFactory=org.apache.lucene.store.NativeFSLockFactory@62ddb62b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 88390 T362 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 88390 T377 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 88390 T330 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 88391 T377 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 88391 T415 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard1_replica2] Registered new searcher Searcher@7d9b17f7 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 88392 T330 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 88391 T350 oasc.ZkController.publish publishing core=awholynewcollection_0_shard1_replica2 state=down
[junit4:junit4] 2> 88392 T330 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 88392 T362 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 88393 T377 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 88393 T392 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 88393 T330 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 88394 T330 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 88393 T362 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 88394 T330 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 88395 T362 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 88395 T330 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 88395 T362 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 88395 T377 oass.SolrIndexSearcher.<init> Opening Searcher@1890a066 main
[junit4:junit4] 2> 88396 T362 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 88396 T362 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 88396 T392 oass.SolrIndexSearcher.<init> Opening Searcher@7a25c2a4 main
[junit4:junit4] 2> 88396 T330 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 88397 T392 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/awholynewcollection_0_shard1_replica3/data/tlog
[junit4:junit4] 2> 88397 T362 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 88396 T377 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/awholynewcollection_0_shard1_replica5/data/tlog
[junit4:junit4] 2> 88398 T392 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 88399 T392 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 88398 T362 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 88399 T330 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 88399 T377 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 88400 T377 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 88399 T362 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 88401 T362 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 88402 T330 oass.SolrIndexSearcher.<init> Opening Searcher@540cf64f main
[junit4:junit4] 2> 88402 T417 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard1_replica3] Registered new searcher Searcher@7a25c2a4 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 88402 T330 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/awholynewcollection_0_shard1_replica4/data/tlog
[junit4:junit4] 2> 88403 T416 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard1_replica5] Registered new searcher Searcher@1890a066 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 88403 T362 oass.SolrIndexSearcher.<init> Opening Searcher@302109de main
[junit4:junit4] 2> 88403 T330 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 88404 T392 oasc.ZkController.publish publishing core=awholynewcollection_0_shard1_replica3 state=down
[junit4:junit4] 2> 88403 T362 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/awholynewcollection_0_shard1_replica1/data/tlog
[junit4:junit4] 2> 88404 T330 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 88405 T362 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 88404 T377 oasc.ZkController.publish publishing core=awholynewcollection_0_shard1_replica5 state=down
[junit4:junit4] 2> 88405 T362 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 88407 T418 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard1_replica4] Registered new searcher Searcher@540cf64f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 88408 T419 oasc.SolrCore.registerSearcher [awholynewcollection_0_shard1_replica1] Registered new searcher Searcher@302109de main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 88409 T330 oasc.ZkController.publish publishing core=awholynewcollection_0_shard1_replica4 state=down
[junit4:junit4] 2> 88409 T362 oasc.ZkController.publish publishing core=awholynewcollection_0_shard1_replica1 state=down
[junit4:junit4] 2> 88951 T337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 88952 T337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:37449_r_em%2Fow_nodes_used_collection_shard1_replica2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"nodes_used_collection_shard1_replica2",
[junit4:junit4] 2> "collection":"nodes_used_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:37449_r_em%2Fow",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:37449/r_em/ow"}
[junit4:junit4] 2> 88957 T337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:53237_r_em%2Fow_nodes_used_collection_shard2_replica1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"nodes_used_collection_shard2_replica1",
[junit4:junit4] 2> "collection":"nodes_used_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53237_r_em%2Fow",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53237/r_em/ow"}
[junit4:junit4] 2> 88960 T337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:34139_r_em%2Fow_nodes_used_collection_shard2_replica2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"nodes_used_collection_shard2_replica2",
[junit4:junit4] 2> "collection":"nodes_used_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:34139_r_em%2Fow",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:34139/r_em/ow"}
[junit4:junit4] 2> 88964 T337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:52867_r_em%2Fow_nodes_used_collection_shard1_replica1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"nodes_used_collection_shard1_replica1",
[junit4:junit4] 2> "collection":"nodes_used_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52867_r_em%2Fow",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52867/r_em/ow"}
[junit4:junit4] 2> 88967 T337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"awholynewcollection_0_shard1_replica2",
[junit4:junit4] 2> "collection":"awholynewcollection_0",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53237_r_em%2Fow",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53237/r_em/ow"}
[junit4:junit4] 2> 88967 T337 oasc.Overseer$ClusterStateUpdater.createCollection Create collection awholynewcollection_0 with numShards 1
[junit4:junit4] 2> 88970 T337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"awholynewcollection_0_shard1_replica3",
[junit4:junit4] 2> "collection":"awholynewcollection_0",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:38602_r_em%2Fow",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38602/r_em/ow"}
[junit4:junit4] 2> 88974 T337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"awholynewcollection_0_shard1_replica5",
[junit4:junit4] 2> "collection":"awholynewcollection_0",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52867_r_em%2Fow",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52867/r_em/ow"}
[junit4:junit4] 2> 88978 T337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"awholynewcollection_0_shard1_replica4",
[junit4:junit4] 2> "collection":"awholynewcollection_0",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:34139_r_em%2Fow",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:34139/r_em/ow"}
[junit4:junit4] 2> 88983 T337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"awholynewcollection_0_shard1_replica1",
[junit4:junit4] 2> "collection":"awholynewcollection_0",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:37449_r_em%2Fow",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:37449/r_em/ow"}
[junit4:junit4] 2> 88989 T413 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 88989 T411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 88989 T336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 88989 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 88990 T382 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 88990 T368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 88990 T354 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 88990 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 89394 T350 oasc.CoreContainer.registerCore registering core: awholynewcollection_0_shard1_replica2
[junit4:junit4] 2> 89395 T350 oasc.ZkController.register Register replica - core:awholynewcollection_0_shard1_replica2 address:http://127.0.0.1:53237/r_em/ow collection:awholynewcollection_0 shard:shard1
[junit4:junit4] 2> 89395 T350 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0/leader_elect/shard1/election
[junit4:junit4] 2> 89403 T350 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 89405 T350 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up: total=5 found=1 timeoutin=180000
[junit4:junit4] 2> 89405 T392 oasc.CoreContainer.registerCore registering core: awholynewcollection_0_shard1_replica3
[junit4:junit4] 2> 89406 T392 oasc.ZkController.register Register replica - core:awholynewcollection_0_shard1_replica3 address:http://127.0.0.1:38602/r_em/ow collection:awholynewcollection_0 shard:shard1
[junit4:junit4] 2> 89406 T377 oasc.CoreContainer.registerCore registering core: awholynewcollection_0_shard1_replica5
[junit4:junit4] 2> 89407 T377 oasc.ZkController.register Register replica - core:awholynewcollection_0_shard1_replica5 address:http://127.0.0.1:52867/r_em/ow collection:awholynewcollection_0 shard:shard1
[junit4:junit4] 2> 89411 T330 oasc.CoreContainer.registerCore registering core: awholynewcollection_0_shard1_replica4
[junit4:junit4] 2> 89411 T362 oasc.CoreContainer.registerCore registering core: awholynewcollection_0_shard1_replica1
[junit4:junit4] 2> 89411 T362 oasc.ZkController.register Register replica - core:awholynewcollection_0_shard1_replica1 address:http://127.0.0.1:37449/r_em/ow collection:awholynewcollection_0 shard:shard1
[junit4:junit4] 2> 89411 T330 oasc.ZkController.register Register replica - core:awholynewcollection_0_shard1_replica4 address:http://127.0.0.1:34139/r_em/ow collection:awholynewcollection_0 shard:shard1
[junit4:junit4] 2> 89661 T348 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:34139_r_em%2Fow_nodes_used_collection_shard2_replica2, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 89661 T348 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:34139_r_em%252Fow_nodes_used_collection_shard2_replica2&state=recovering&nodeName=127.0.0.1:34139_r_em%252Fow&action=PREPRECOVERY&checkLive=true&core=nodes_used_collection_shard2_replica1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2001
[junit4:junit4] 2> 89663 T363 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:52867_r_em%2Fow_nodes_used_collection_shard1_replica1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 89663 T363 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:52867_r_em%252Fow_nodes_used_collection_shard1_replica1&state=recovering&nodeName=127.0.0.1:52867_r_em%252Fow&action=PREPRECOVERY&checkLive=true&core=nodes_used_collection_shard1_replica2&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2001
[junit4:junit4] 2> 89906 T350 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 89907 T350 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 89907 T350 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53237/r_em/ow/awholynewcollection_0_shard1_replica2/
[junit4:junit4] 2> 89907 T350 oasu.PeerSync.sync PeerSync: core=awholynewcollection_0_shard1_replica2 url=http://127.0.0.1:53237/r_em/ow START replicas=[http://127.0.0.1:38602/r_em/ow/awholynewcollection_0_shard1_replica3/, http://127.0.0.1:52867/r_em/ow/awholynewcollection_0_shard1_replica5/, http://127.0.0.1:34139/r_em/ow/awholynewcollection_0_shard1_replica4/, http://127.0.0.1:37449/r_em/ow/awholynewcollection_0_shard1_replica1/] nUpdates=100
[junit4:junit4] 2> 89908 T350 oasu.PeerSync.sync PeerSync: core=awholynewcollection_0_shard1_replica2 url=http://127.0.0.1:53237/r_em/ow DONE. We have no versions. sync failed.
[junit4:junit4] 2> 89909 T350 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next canidate
[junit4:junit4] 2> 89909 T350 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4:junit4] 2> 89910 T350 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53237/r_em/ow/awholynewcollection_0_shard1_replica2/
[junit4:junit4] 2> 89910 T350 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_0/leaders/shard1
[junit4:junit4] 2> ASYNC NEW_CORE C82 name=awholynewcollection_0_shard1_replica5 org.apache.solr.core.SolrCore@13190ef8 url=http://127.0.0.1:52867/r_em/ow/awholynewcollection_0_shard1_replica5 node=127.0.0.1:52867_r_em%2Fow C82_STATE=coll:awholynewcollection_0 core:awholynewcollection_0_shard1_replica5 props:{shard=shard1, state=down, core=awholynewcollection_0_shard1_replica5, collection=awholynewcollection_0, node_name=127.0.0.1:52867_r_em%2Fow, base_url=http://127.0.0.1:52867/r_em/ow}
[junit4:junit4] 2> 89915 T378 C82 P52867 oasc.SolrCore.execute [awholynewcollection_0_shard1_replica5] webapp=/r_em/ow path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> ASYNC NEW_CORE C83 name=awholynewcollection_0_shard1_replica4 org.apache.solr.core.SolrCore@6c0d6ed4 url=http://127.0.0.1:34139/r_em/ow/awholynewcollection_0_shard1_replica4 node=127.0.0.1:34139_r_em%2Fow C83_STATE=coll:awholynewcollection_0 core:awholynewcollection_0_shard1_replica4 props:{shard=shard1, state=down, core=awholynewcollection_0_shard1_replica4, collection=awholynewcollection_0, node_name=127.0.0.1:34139_r_em%2Fow, base_url=http://127.0.0.1:34139/r_em/ow}
[junit4:junit4] 2> 89916 T329 C83 P34139 oasc.SolrCore.execute [awholynewcollection_0_shard1_replica4] webapp=/r_em/ow path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> ASYNC NEW_CORE C84 name=awholynewcollection_0_shard1_replica1 org.apache.solr.core.SolrCore@3efee18b url=http://127.0.0.1:37449/r_em/ow/awholynewcollection_0_shard1_replica1 node=127.0.0.1:37449_r_em%2Fow C84_STATE=coll:awholynewcollection_0 core:awholynewcollection_0_shard1_replica1 props:{shard=shard1, state=down, core=awholynewcollection_0_shard1_replica1, collection=awholynewcollection_0, node_name=127.0.0.1:37449_r_em%2Fow, base_url=http://127.0.0.1:37449/r_em/ow}
[junit4:junit4] 2> 89916 T363 C84 P37449 oasc.SolrCore.execute [awholynewcollection_0_shard1_replica1] webapp=/r_em/ow path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> ASYNC NEW_CORE C85 name=awholynewcollection_0_shard1_replica3 org.apache.solr.core.SolrCore@15329471 url=http://127.0.0.1:38602/r_em/ow/awholynewcollection_0_shard1_replica3 node=127.0.0.1:38602_r_em%2Fow C85_STATE=coll:awholynewcollection_0 core:awholynewcollection_0_shard1_replica3 props:{shard=shard1, state=down, core=awholynewcollection_0_shard1_replica3, collection=awholynewcollection_0, node_name=127.0.0.1:38602_r_em%2Fow, base_url=http://127.0.0.1:38602/r_em/ow}
[junit4:junit4] 2> 89916 T389 C85 P38602 oasc.SolrCore.execute [awholynewcollection_0_shard1_replica3] webapp=/r_em/ow path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 90512 T337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 90521 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 90521 T368 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 90521 T382 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 90521 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 90521 T413 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 90521 T411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 90521 T336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 90521 T354 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 90560 T392 oasc.ZkController.register We are http://127.0.0.1:38602/r_em/ow/awholynewcollection_0_shard1_replica3/ and leader is http://127.0.0.1:53237/r_em/ow/awholynewcollection_0_shard1_replica2/
[junit4:junit4] 2> 90561 T377 oasc.ZkController.register We are http://127.0.0.1:52867/r_em/ow/awholynewcollection_0_shard1_replica5/ and leader is http://127.0.0.1:53237/r_em/ow/awholynewcollection_0_shard1_replica2/
[junit4:junit4] 2> 90561 T377 oasc.ZkController.register No LogReplay needed for core=awholynewcollection_0_shard1_replica5 baseURL=http://127.0.0.1:52867/r_em/ow
[junit4:junit4] 2> 90562 T377 oasc.ZkController.checkRecovery Core needs to recover:awholynewcollection_0_shard1_replica5
[junit4:junit4] 2> 90562 T377 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> 90561 T392 oasc.ZkController.register No LogReplay needed for core=awholynewcollection_0_shard1_replica3 baseURL=http://127.0.0.1:38602/r_em/ow
[junit4:junit4] 2> 90563 T392 oasc.ZkController.checkRecovery Core needs to recover:awholynewcollection_0_shard1_replica3
[junit4:junit4] 2> 90563 T392 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> 90564 T377 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 90563 T424 C82 P52867 oasc.RecoveryStrategy.run Starting recovery process. core=awholynewcollection_0_shard1_replica5 recoveringAfterStartup=true
[junit4:junit4] 2> 90564 T392 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 90564 T425 C85 P38602 oasc.RecoveryStrategy.run Starting recovery process. core=awholynewcollection_0_shard1_replica3 recoveringAfterStartup=true
[junit4:junit4] 2> 90564 T424 C82 P52867 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 90565 T377 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=1&shard=shard1&name=awholynewcollection_0_shard1_replica5&action=CREATE&collection=awholynewcollection_0&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=2868
[junit4:junit4] 2> 90565 T425 C85 P38602 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 90565 T392 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=1&shard=shard1&name=awholynewcollection_0_shard1_replica3&action=CREATE&collection=awholynewcollection_0&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=2869
[junit4:junit4] 2> 90565 T424 C82 P52867 oasc.ZkController.publish publishing core=awholynewcollection_0_shard1_replica5 state=recovering
[junit4:junit4] 2> 90566 T425 C85 P38602 oasc.ZkController.publish publishing core=awholynewcollection_0_shard1_replica3 state=recovering
[junit4:junit4] 2> 90567 T362 oasc.ZkController.register We are http://127.0.0.1:37449/r_em/ow/awholynewcollection_0_shard1_replica1/ and leader is http://127.0.0.1:53237/r_em/ow/awholynewcollection_0_shard1_replica2/
[junit4:junit4] 2> 90567 T330 oasc.ZkController.register We are http://127.0.0.1:34139/r_em/ow/awholynewcollection_0_shard1_replica4/ and leader is http://127.0.0.1:53237/r_em/ow/awholynewcollection_0_shard1_replica2/
[junit4:junit4] 2> 90567 T362 oasc.ZkController.register No LogReplay needed for core=awholynewcollection_0_shard1_replica1 baseURL=http://127.0.0.1:37449/r_em/ow
[junit4:junit4] 2> 90567 T362 oasc.ZkController.checkRecovery Core needs to recover:awholynewcollection_0_shard1_replica1
[junit4:junit4] 2> 90568 T362 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> 90567 T330 oasc.ZkController.register No LogReplay needed for core=awholynewcollection_0_shard1_replica4 baseURL=http://127.0.0.1:34139/r_em/ow
[junit4:junit4] 2> 90568 T330 oasc.ZkController.checkRecovery Core needs to recover:awholynewcollection_0_shard1_replica4
[junit4:junit4] 2> 90568 T330 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> 90568 T426 C84 P37449 oasc.RecoveryStrategy.run Starting recovery process. core=awholynewcollection_0_shard1_replica1 recoveringAfterStartup=true
[junit4:junit4] 2> 90568 T425 C85 P38602 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 90567 T424 C82 P52867 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 90569 T427 C83 P34139 oasc.RecoveryStrategy.run Starting recovery process. core=awholynewcollection_0_shard1_replica4 recoveringAfterStartup=true
[junit4:junit4] 2> 90569 T330 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 90569 T426 C84 P37449 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 90569 T362 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 90569 T427 C83 P34139 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 90571 T330 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=1&shard=shard1&name=awholynewcollection_0_shard1_replica4&action=CREATE&collection=awholynewcollection_0&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=2873
[junit4:junit4] 2> 90571 T362 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=1&shard=shard1&name=awholynewcollection_0_shard1_replica1&action=CREATE&collection=awholynewcollection_0&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=2873
[junit4:junit4] 2> 90572 T426 C84 P37449 oasc.ZkController.publish publishing core=awholynewcollection_0_shard1_replica1 state=recovering
[junit4:junit4] 2> 90572 T427 C83 P34139 oasc.ZkController.publish publishing core=awholynewcollection_0_shard1_replica4 state=recovering
[junit4:junit4] 2> 90572 T350 oasc.ZkController.register We are http://127.0.0.1:53237/r_em/ow/awholynewcollection_0_shard1_replica2/ and leader is http://127.0.0.1:53237/r_em/ow/awholynewcollection_0_shard1_replica2/
[junit4:junit4] 2> 90576 T427 C83 P34139 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 90576 T350 oasc.ZkController.register No LogReplay needed for core=awholynewcollection_0_shard1_replica2 baseURL=http://127.0.0.1:53237/r_em/ow
[junit4:junit4] 2> 90577 T350 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 90576 T348 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:38602_r_em%2Fow_awholynewcollection_0_shard1_replica3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 90574 T426 C84 P37449 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 90577 T350 oasc.ZkController.publish publishing core=awholynewcollection_0_shard1_replica2 state=active
[junit4:junit4] 2> 90577 T349 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:52867_r_em%2Fow_awholynewcollection_0_shard1_replica5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 90579 T350 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 90581 T350 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={numShards=1&shard=shard1&name=awholynewcollection_0_shard1_replica2&action=CREATE&collection=awholynewcollection_0&wt=javabin&qt=/admin/cores&version=2} status=0 QTime=2884
[junit4:junit4] 2> 90582 T338 oasc.OverseerCollectionProcessor.createCollection Finished create command on all shards for collection: awholynewcollection_0
[junit4:junit4] 2> 90583 T429 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:34139_r_em%2Fow_awholynewcollection_0_shard1_replica4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 90586 T354 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work/qnr-0000000002 state: SyncConnected type NodeDataChanged
[junit4:junit4] 2> 90587 T350 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:37449_r_em%2Fow_awholynewcollection_0_shard1_replica1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 90588 T336 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4] 2> 90588 T338 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Message id:/overseer/collection-queue-work/qn-0000000002 complete, response:{success={null={responseHeader={status=0,QTime=2868},core=awholynewcollection_0_shard1_replica5},null={responseHeader={status=0,QTime=2869},core=awholynewcollection_0_shard1_replica3},null={responseHeader={status=0,QTime=2873},core=awholynewcollection_0_shard1_replica4},null={responseHeader={status=0,QTime=2873},core=awholynewcollection_0_shard1_replica1},null={responseHeader={status=0,QTime=2884},core=awholynewcollection_0_shard1_replica2}}}
[junit4:junit4] 2> 90590 T347 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={numShards=1&maxShardsPerNode=2&name=awholynewcollection_0&replicationFactor=5&action=CREATE&wt=javabin&version=2} status=0 QTime=2900
[junit4:junit4] 2> 90593 T314 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 90594 T314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 90596 T431 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63711742 name:ZooKeeperConnection Watcher:127.0.0.1:33865/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 90596 T314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 90598 T314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 90604 T347 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=4&maxShardsPerNode=5&name=awholynewcollection_1&replicationFactor=5&action=CREATE&wt=javabin&version=2
[junit4:junit4] 2> 90605 T336 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
[junit4:junit4] 2> 90606 T338 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000004 message:{
[junit4:junit4] 2> "operation":"createcollection",
[junit4:junit4] 2> "numShards":"4",
[junit4:junit4] 2> "maxShardsPerNode":"5",
[junit4:junit4] 2> "createNodeSet":null,
[junit4:junit4] 2> "name":"awholynewcollection_1",
[junit4:junit4] 2> "replicationFactor":"5"}
[junit4:junit4] 2> 90606 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard1_replica1 as part of slice shard1 of collection awholynewcollection_1 on 127.0.0.1:37449_r_em%2Fow
[junit4:junit4] 2> 90607 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard1_replica2 as part of slice shard1 of collection awholynewcollection_1 on 127.0.0.1:52867_r_em%2Fow
[junit4:junit4] 2> 90607 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard1_replica3 as part of slice shard1 of collection awholynewcollection_1 on 127.0.0.1:38602_r_em%2Fow
[junit4:junit4] 2> 90607 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard1_replica4 as part of slice shard1 of collection awholynewcollection_1 on 127.0.0.1:34139_r_em%2Fow
[junit4:junit4] 2> 90607 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard1_replica5 as part of slice shard1 of collection awholynewcollection_1 on 127.0.0.1:53237_r_em%2Fow
[junit4:junit4] 2> 90607 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard2_replica1 as part of slice shard2 of collection awholynewcollection_1 on 127.0.0.1:37449_r_em%2Fow
[junit4:junit4] 2> 90608 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard2_replica2 as part of slice shard2 of collection awholynewcollection_1 on 127.0.0.1:52867_r_em%2Fow
[junit4:junit4] 2> 90608 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard2_replica3 as part of slice shard2 of collection awholynewcollection_1 on 127.0.0.1:38602_r_em%2Fow
[junit4:junit4] 2> 90608 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard2_replica4 as part of slice shard2 of collection awholynewcollection_1 on 127.0.0.1:34139_r_em%2Fow
[junit4:junit4] 2> 90609 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard2_replica5 as part of slice shard2 of collection awholynewcollection_1 on 127.0.0.1:53237_r_em%2Fow
[junit4:junit4] 2> 90609 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard3_replica1 as part of slice shard3 of collection awholynewcollection_1 on 127.0.0.1:37449_r_em%2Fow
[junit4:junit4] 2> 90610 T363 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_1_shard1_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/awholynewcollection_1_shard1_replica1
[junit4:junit4] 2> 90610 T437 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_1_shard1_replica5' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/awholynewcollection_1_shard1_replica5
[junit4:junit4] 2> 90610 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard3_replica2 as part of slice shard3 of collection awholynewcollection_1 on 127.0.0.1:52867_r_em%2Fow
[junit4:junit4] 2> 90610 T363 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_1
[junit4:junit4] 2> 90611 T437 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_1
[junit4:junit4] 2> 90612 T363 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_1
[junit4:junit4] 2> 90612 T363 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 90613 T331 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_1_shard2_replica4' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/awholynewcollection_1_shard2_replica4
[junit4:junit4] 2> 90614 T331 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_1
[junit4:junit4] 2> 90614 T362 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_1_shard3_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/awholynewcollection_1_shard3_replica1
[junit4:junit4] 2> 90614 T362 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_1
[junit4:junit4] 2> 90615 T361 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_1_shard2_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/awholynewcollection_1_shard2_replica1
[junit4:junit4] 2> 90616 T361 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_1
[junit4:junit4] 2> 90613 T391 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_1_shard2_replica3' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/awholynewcollection_1_shard2_replica3
[junit4:junit4] 2> 90617 T391 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_1
[junit4:junit4] 2> 90618 T363 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 90613 T390 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_1_shard1_replica3' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/awholynewcollection_1_shard1_replica3
[junit4:junit4] 2> 90618 T390 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_1
[junit4:junit4] 2> 90613 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard3_replica3 as part of slice shard3 of collection awholynewcollection_1 on 127.0.0.1:38602_r_em%2Fow
[junit4:junit4] 2> 90612 T376 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_1_shard1_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/awholynewcollection_1_shard1_replica2
[junit4:junit4] 2> 90619 T390 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_1
[junit4:junit4] 2> 90612 T332 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_1_shard1_replica4' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/awholynewcollection_1_shard1_replica4
[junit4:junit4] 2> 90620 T332 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_1
[junit4:junit4] 2> 90612 T437 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_1
[junit4:junit4] 2> 90621 T332 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_1
[junit4:junit4] 2> 90620 T390 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 90619 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard3_replica4 as part of slice shard3 of collection awholynewcollection_1 on 127.0.0.1:34139_r_em%2Fow
[junit4:junit4] 2> 90619 T376 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_1
[junit4:junit4] 2> 90618 T391 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_1
[junit4:junit4] 2> 90623 T391 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 90618 T363 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_1
[junit4:junit4] 2> 90618 T362 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_1
[junit4:junit4] 2> 90625 T362 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 90625 T391 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 90618 T361 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_1
[junit4:junit4] 2> 90618 T377 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_1_shard3_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/awholynewcollection_1_shard3_replica2
[junit4:junit4] 2> 90626 T377 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_1
[junit4:junit4] 2> 90617 T331 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_1
[junit4:junit4] 2> 90615 T439 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_1_shard2_replica5' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/awholynewcollection_1_shard2_replica5
[junit4:junit4] 2> 90627 T439 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_1
[junit4:junit4] 2> 90614 T375 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_1_shard2_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/awholynewcollection_1_shard2_replica2
[junit4:junit4] 2> 90627 T375 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_1
[junit4:junit4] 2> 90627 T363 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_1
[junit4:junit4] 2> 90627 T377 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 90628 T377 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_1
[junit4:junit4] 2> 90626 T362 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 90626 T331 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 90625 T361 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 90625 T391 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_1
[junit4:junit4] 2> 90630 T377 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/awholynewcollection_1_shard3_replica2/'
[junit4:junit4] 2> 90625 T330 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_1_shard3_replica4' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/awholynewcollection_1_shard3_replica4
[junit4:junit4] 2> 90630 T330 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_1
[junit4:junit4] 2> 90624 T376 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:awholynewcollection_1
[junit4:junit4] 2> 90624 T390 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 90623 T389 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_1_shard3_replica3' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1369623830663/awholynewcollection_1_shard3_replica3
[junit4:junit4] 2> 90632 T389 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_1
[junit4:junit4] 2> 90622 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard3_replica5 as part of slice shard3 of collection awholynewcollection_1 on 127.0.0.1:53237_r_em%2Fow
[junit4:junit4] 2> 90622 T332 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 90622 T437 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 90632 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard4_replica1 as part of slice shard4 of collection awholynewcollection_1 on 127.0.0.1:37449_r_em%2Fow
[junit4:junit4] 2> 90631 T390 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_1
[junit4:junit4] 2> 90631 T376 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 90630 T363 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1369623828302/awholynewcollection_1_shard1_replica1/'
[junit4:junit4] 2> 90629 T362 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_1
[junit4:junit4] 2> 90635 T437 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_1
[junit4:junit4] 2> 90628 T375 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 90636 T375 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_1
[junit4:junit4] 2> 90627 T439 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 90638 T439 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_1
[junit4:junit4] 2> 90636 T376 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_1
[junit4:junit4] 2> 90635 T332 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_1
[junit4:junit4] 2> 90635 T361 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_1
[junit4:junit4] 2> 90633 T389 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 90644 T389 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_1
[junit4:junit4] 2> 90633 T330 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 90644 T330 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_1
[junit4:junit4] 2> 90645 T437 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_1
[junit4:junit4] 2> 90645 T445 oasc.CoreContainer.create Creating SolrCore 'awholynewcollection_1_shard3_replica5' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/awholynewcollection_1_shard3_replica5
[junit4:junit4] 2> 90646 T445 oasc.ZkController.createCollectionZkNode Check for collection zkNode:awholynewcollection_1
[junit4:junit4] 2> 90633 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard4_replica2 as part of slice shard4 of collection awholynewcollection_1 on 127.0.0.1:52867_r_em%2Fow
[junit4:junit4] 2> 90647 T362 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_1
[junit4:junit4] 2> 90633 T331 oascc.SolrZkClient.makePath makePath: /collections/awholynewcollection_1
[junit4:junit4] 2> 90647 T338 oasc.OverseerCollectionProcessor.createCollection Creating shard awholynewcollection_1_shard4_replica3 as part of slice shard4 of collection awholynewcollection_1 on 127.0.0.1:38602_r_em%2Fow
[junit4:junit4] 2> 90646 T375 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1369623829477/awholynewcollection_1_shard2_replica2/'
[junit4:junit4] 2> 90648 T390 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_1
[junit4:junit4] 2> 90648 T330 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1369623826846/awholynewcollection_1_shard3_replica4/'
[junit4:junit4] 2> 90646 T439 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1369623827088/awholynewcollection_1_shard2_replica5/'
[junit4:junit4] 2> 90644 T391 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_1
[junit4:junit4] 2> 90656 T361 oasc.ZkController.readConfigName Load collection config from:/collections/awholynewcollection_1
[junit4:junit4]
[...truncated too long message...]
overseer_elect/election/89759667099140108-127.0.0.1:38602_r_em%2Fow-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89759667099140104-127.0.0.1:37449_r_em%2Fow-n_0000000002 (0)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89759667099140099-127.0.0.1:34139_r_em%2Fow-n_0000000000"}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=B87450A4B3E529F4 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=no_NO_NY -Dtests.timezone=Asia/Almaty -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE 283s J1 | CollectionsAPIDistributedZkTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: There are still nodes recoverying - waited for 230 seconds
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([B87450A4B3E529F4:3992DEBCC4BA49C8]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:173)
[junit4:junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
[junit4:junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
[junit4:junit4] > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:512)
[junit4:junit4] > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:146)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 362892 T314 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 283487 T313 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
[junit4:junit4] 2> 364073 T396 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_2/leader_elect/shard3/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> 364074 T396 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_2/leader_elect/shard1/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> 364074 T396 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/awholynewcollection_2/leader_elect/shard2/election
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4] 2>
[junit4:junit4] 2> 364075 T396 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 364075 T396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 364075 T396 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {timestamp=MockVariableIntBlock(baseBlockSize=6), range_facet_l=MockVariableIntBlock(baseBlockSize=6), id=PostingsFormat(name=Memory doPackFST= true), range_facet_sl=PostingsFormat(name=SimpleText), a_t=MockVariableIntBlock(baseBlockSize=6), text=PostingsFormat(name=SimpleText), range_facet_si=Pulsing41(freqCutoff=3 minBlockSize=67 maxBlockSize=146), _version_=MockVariableIntBlock(baseBlockSize=6), other_tl1=MockVariableIntBlock(baseBlockSize=6), multiDefault=Pulsing41(freqCutoff=3 minBlockSize=67 maxBlockSize=146), a_si=Pulsing41(freqCutoff=3 minBlockSize=67 maxBlockSize=146), intDefault=MockVariableIntBlock(baseBlockSize=6)}, docValues:{timestamp=DocValuesFormat(name=Asserting)}, sim=DefaultSimilarity, locale=no_NO_NY, timezone=Asia/Almaty
[junit4:junit4] 2> NOTE: Linux 3.2.0-41-generic amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=8,threads=1,free=74933080,total=302383104
[junit4:junit4] 2> NOTE: All tests run in this JVM: [SuggesterTSTTest, TestLMJelinekMercerSimilarityFactory, DisMaxRequestHandlerTest, HighlighterConfigTest, IndexReaderFactoryTest, ClusterStateTest, IndexBasedSpellCheckerTest, TestDefaultSearchFieldResource, TestRTGBase, TestZkChroot, TestOmitPositions, SpellCheckCollatorTest, DirectUpdateHandlerTest, EchoParamsTest, OutputWriterTest, TestRangeQuery, HighlighterTest, StandardRequestHandlerTest, TestSchemaVersionResource, FileUtilsTest, RequestHandlersTest, TestSolrQueryParserDefaultOperatorResource, TestReload, TestSolr4Spatial, TestFastOutputStream, TestDFRSimilarityFactory, TestLazyCores, DefaultValueUpdateProcessorTest, TestCollationField, TestCopyFieldCollectionResource, TestLuceneMatchVersion, URLClassifyProcessorTest, CollectionsAPIDistributedZkTest]
[junit4:junit4] Completed on J1 in 285.27s, 1 test, 1 failure <<< FAILURES!
[...truncated 629 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:380: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:360: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1240: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:884: There were test failures: 295 suites, 1231 tests, 1 failure, 13 ignored (7 assumptions)
Total time: 40 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure