You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/28 09:22:22 UTC
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_25) - Build #
6346 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6346/
Java: 64bit/jdk1.7.0_25 -XX:-UseCompressedOops -XX:+UseSerialGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:39944/sr/collection1lastClient and got 255 from http://127.0.0.1:50010/sr/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:39944/sr/collection1lastClient and got 255 from http://127.0.0.1:50010/sr/collection1
at __randomizedtesting.SeedInfo.seed([C103F4B8ABD96C0D:40E57AA0DC860C31]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1045)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:237)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9660 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4] 2> 664247 T2165 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /sr/
[junit4:junit4] 2> 664251 T2165 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1372403597234
[junit4:junit4] 2> 664252 T2165 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 664252 T2166 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 664352 T2165 oasc.ZkTestServer.run start zk server on port:42941
[junit4:junit4] 2> 664354 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 664577 T2172 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4775b87d name:ZooKeeperConnection Watcher:127.0.0.1:42941 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 664578 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 664578 T2165 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 664598 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 664600 T2174 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15f44133 name:ZooKeeperConnection Watcher:127.0.0.1:42941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 664601 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 664601 T2165 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 664606 T2165 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 664610 T2165 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 664614 T2165 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 664617 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 664618 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 664626 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 664626 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 664632 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 664632 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 664637 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 664638 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 664642 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 664642 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 664647 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 664648 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 664652 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 664653 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 664658 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 664658 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 664662 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 664663 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 664746 T2165 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 664748 T2165 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40343
[junit4:junit4] 2> 664749 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 664749 T2165 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 664749 T2165 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651
[junit4:junit4] 2> 664749 T2165 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651/solr.xml
[junit4:junit4] 2> 664750 T2165 oasc.CoreContainer.<init> New CoreContainer 1935753536
[junit4:junit4] 2> 664750 T2165 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651/'
[junit4:junit4] 2> 664750 T2165 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651/'
[junit4:junit4] 2> 664818 T2165 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 664819 T2165 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 664819 T2165 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 664820 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 664820 T2165 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 664820 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 664820 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 664821 T2165 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 664821 T2165 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 664822 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 664825 T2165 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 664826 T2165 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42941/solr
[junit4:junit4] 2> 664826 T2165 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 664827 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 664828 T2185 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c8e6076 name:ZooKeeperConnection Watcher:127.0.0.1:42941 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 664829 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 664830 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 664834 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 664835 T2187 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@662e57d8 name:ZooKeeperConnection Watcher:127.0.0.1:42941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 664835 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 664836 T2165 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 664839 T2165 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 664841 T2165 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 664843 T2165 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40343_sr
[junit4:junit4] 2> 664844 T2165 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40343_sr
[junit4:junit4] 2> 664847 T2165 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 664851 T2165 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 664854 T2165 oasc.Overseer.start Overseer (id=89941842148524035-127.0.0.1:40343_sr-n_0000000000) starting
[junit4:junit4] 2> 664860 T2165 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 664865 T2189 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 664865 T2165 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 664867 T2165 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 664869 T2165 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 664872 T2188 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 664874 T2190 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 664875 T2190 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 664876 T2190 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 666375 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 666376 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40343_sr",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40343/sr"}
[junit4:junit4] 2> 666376 T2188 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 666377 T2188 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 666381 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 666876 T2190 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 666877 T2190 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651/collection1
[junit4:junit4] 2> 666877 T2190 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 666878 T2190 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 666878 T2190 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 666880 T2190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651/collection1/'
[junit4:junit4] 2> 666881 T2190 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651/collection1/lib/README' to classloader
[junit4:junit4] 2> 666881 T2190 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 666917 T2190 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 666958 T2190 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 666960 T2190 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 666964 T2190 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 667491 T2190 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 667496 T2190 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 667498 T2190 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 667534 T2190 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 667537 T2190 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 667541 T2190 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 667543 T2190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 667544 T2190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 667544 T2190 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 667546 T2190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 667547 T2190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 667547 T2190 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 667547 T2190 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372403597233/control/data/
[junit4:junit4] 2> 667548 T2190 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b71c166
[junit4:junit4] 2> 667549 T2190 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/control/data
[junit4:junit4] 2> 667550 T2190 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372403597233/control/data/index/
[junit4:junit4] 2> 667550 T2190 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372403597233/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 667551 T2190 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/control/data/index
[junit4:junit4] 2> 667555 T2190 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@599cdbaf lockFactory=org.apache.lucene.store.NativeFSLockFactory@475f075f),segFN=segments_1,generation=1}
[junit4:junit4] 2> 667556 T2190 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 667559 T2190 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 667559 T2190 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 667560 T2190 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 667560 T2190 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 667561 T2190 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 667561 T2190 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 667561 T2190 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 667562 T2190 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 667562 T2190 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 667566 T2190 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 667568 T2190 oass.SolrIndexSearcher.<init> Opening Searcher@7b2c6780 main
[junit4:junit4] 2> 667569 T2190 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 667569 T2190 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 667572 T2191 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7b2c6780 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 667572 T2190 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 667573 T2190 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40343/sr collection:control_collection shard:shard1
[junit4:junit4] 2> 667573 T2190 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 667579 T2190 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4] 2> 667582 T2190 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 667582 T2190 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 667582 T2190 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40343/sr/collection1/
[junit4:junit4] 2> 667583 T2190 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 667583 T2190 oasc.SyncStrategy.syncToMe http://127.0.0.1:40343/sr/collection1/ has no replicas
[junit4:junit4] 2> 667583 T2190 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40343/sr/collection1/ shard1
[junit4:junit4] 2> 667584 T2190 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 667885 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 667893 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 667901 T2190 oasc.ZkController.register We are http://127.0.0.1:40343/sr/collection1/ and leader is http://127.0.0.1:40343/sr/collection1/
[junit4:junit4] 2> 667901 T2190 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40343/sr
[junit4:junit4] 2> 667901 T2190 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 667901 T2190 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 667902 T2190 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 667904 T2190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 667915 T2165 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 667915 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 667916 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 667921 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 667926 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 667933 T2194 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b69efd2 name:ZooKeeperConnection Watcher:127.0.0.1:42941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 667933 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 667934 T2165 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 667936 T2165 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 668041 T2165 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 668043 T2165 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39944
[junit4:junit4] 2> 668044 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 668045 T2165 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 668045 T2165 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918
[junit4:junit4] 2> 668046 T2165 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918/solr.xml
[junit4:junit4] 2> 668046 T2165 oasc.CoreContainer.<init> New CoreContainer 528298614
[junit4:junit4] 2> 668046 T2165 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918/'
[junit4:junit4] 2> 668047 T2165 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918/'
[junit4:junit4] 2> 668091 T2165 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 668091 T2165 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 668092 T2165 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 668092 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 668092 T2165 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 668092 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 668093 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 668093 T2165 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 668093 T2165 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 668094 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 668097 T2165 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 668097 T2165 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42941/solr
[junit4:junit4] 2> 668098 T2165 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 668098 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 668099 T2205 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a347ce4 name:ZooKeeperConnection Watcher:127.0.0.1:42941 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 668100 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 668101 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 668104 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 668105 T2207 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5dadfdec name:ZooKeeperConnection Watcher:127.0.0.1:42941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 668106 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 668108 T2165 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 669110 T2165 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39944_sr
[junit4:junit4] 2> 669112 T2165 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39944_sr
[junit4:junit4] 2> 669114 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 669115 T2207 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 669115 T2194 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 669119 T2187 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 669124 T2208 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 669124 T2208 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 669125 T2208 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 669397 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 669398 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40343_sr",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40343/sr"}
[junit4:junit4] 2> 669401 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39944_sr",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39944/sr"}
[junit4:junit4] 2> 669401 T2188 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 669401 T2188 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 669405 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 669405 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 669405 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 670126 T2208 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 670126 T2208 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918/collection1
[junit4:junit4] 2> 670126 T2208 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 670127 T2208 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 670127 T2208 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 670128 T2208 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918/collection1/'
[junit4:junit4] 2> 670129 T2208 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918/collection1/lib/README' to classloader
[junit4:junit4] 2> 670129 T2208 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 670163 T2208 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 670192 T2208 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 670193 T2208 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 670196 T2208 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 670530 T2208 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 670535 T2208 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 670537 T2208 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 670547 T2208 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 670550 T2208 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 670561 T2208 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 670562 T2208 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 670562 T2208 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 670563 T2208 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 670563 T2208 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 670564 T2208 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 670564 T2208 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 670564 T2208 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1/
[junit4:junit4] 2> 670564 T2208 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b71c166
[junit4:junit4] 2> 670565 T2208 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1
[junit4:junit4] 2> 670565 T2208 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1/index/
[junit4:junit4] 2> 670565 T2208 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 670565 T2208 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1/index
[junit4:junit4] 2> 670567 T2208 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@673bbff8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2ee94c06),segFN=segments_1,generation=1}
[junit4:junit4] 2> 670567 T2208 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 670568 T2208 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 670569 T2208 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 670569 T2208 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 670570 T2208 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 670570 T2208 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 670570 T2208 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 670570 T2208 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 670571 T2208 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 670571 T2208 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 670574 T2208 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 670576 T2208 oass.SolrIndexSearcher.<init> Opening Searcher@2bcf32cc main
[junit4:junit4] 2> 670577 T2208 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 670577 T2208 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 670580 T2209 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2bcf32cc main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 670581 T2208 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 670581 T2208 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39944/sr collection:collection1 shard:shard1
[junit4:junit4] 2> 670582 T2208 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 670587 T2208 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4] 2> 670590 T2208 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 670590 T2208 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 670590 T2208 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39944/sr/collection1/
[junit4:junit4] 2> 670590 T2208 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 670590 T2208 oasc.SyncStrategy.syncToMe http://127.0.0.1:39944/sr/collection1/ has no replicas
[junit4:junit4] 2> 670590 T2208 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39944/sr/collection1/ shard1
[junit4:junit4] 2> 670591 T2208 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 670910 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 670917 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 670917 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 670917 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 670946 T2208 oasc.ZkController.register We are http://127.0.0.1:39944/sr/collection1/ and leader is http://127.0.0.1:39944/sr/collection1/
[junit4:junit4] 2> 670946 T2208 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39944/sr
[junit4:junit4] 2> 670947 T2208 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 670947 T2208 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 670947 T2208 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 670949 T2208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 670950 T2165 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 670950 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 670951 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 671016 T2165 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 671018 T2165 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53322
[junit4:junit4] 2> 671018 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 671019 T2165 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 671019 T2165 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936
[junit4:junit4] 2> 671019 T2165 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936/solr.xml
[junit4:junit4] 2> 671019 T2165 oasc.CoreContainer.<init> New CoreContainer 461621668
[junit4:junit4] 2> 671020 T2165 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936/'
[junit4:junit4] 2> 671020 T2165 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936/'
[junit4:junit4] 2> 671066 T2165 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 671066 T2165 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 671066 T2165 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 671067 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 671067 T2165 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 671068 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 671068 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 671068 T2165 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 671069 T2165 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 671069 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 671072 T2165 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 671073 T2165 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42941/solr
[junit4:junit4] 2> 671073 T2165 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 671074 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 671075 T2221 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5508c842 name:ZooKeeperConnection Watcher:127.0.0.1:42941 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 671076 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 671077 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 671082 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 671083 T2223 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7afd05c2 name:ZooKeeperConnection Watcher:127.0.0.1:42941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 671083 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 671087 T2165 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 672091 T2165 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53322_sr
[junit4:junit4] 2> 672093 T2165 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53322_sr
[junit4:junit4] 2> 672097 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 672097 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 672098 T2194 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 672098 T2223 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 672098 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 672101 T2187 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 672101 T2207 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 672106 T2224 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 672106 T2224 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 672108 T2224 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 672424 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 672425 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39944_sr",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39944/sr"}
[junit4:junit4] 2> 672430 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53322_sr",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53322/sr"}
[junit4:junit4] 2> 672431 T2188 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 672431 T2188 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 672436 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 672436 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 672436 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 672436 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 673108 T2224 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 673109 T2224 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936/collection1
[junit4:junit4] 2> 673109 T2224 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 673110 T2224 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 673111 T2224 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 673113 T2224 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936/collection1/'
[junit4:junit4] 2> 673114 T2224 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936/collection1/lib/README' to classloader
[junit4:junit4] 2> 673114 T2224 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 673154 T2224 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 673211 T2224 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 673213 T2224 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 673216 T2224 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 673597 T2224 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 673602 T2224 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 673604 T2224 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 673621 T2224 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 673625 T2224 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 673629 T2224 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 673631 T2224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 673631 T2224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 673632 T2224 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 673633 T2224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 673634 T2224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 673634 T2224 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 673634 T2224 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty2/
[junit4:junit4] 2> 673635 T2224 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b71c166
[junit4:junit4] 2> 673635 T2224 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty2
[junit4:junit4] 2> 673636 T2224 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty2/index/
[junit4:junit4] 2> 673636 T2224 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 673636 T2224 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty2/index
[junit4:junit4] 2> 673639 T2224 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60317470 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48e612de),segFN=segments_1,generation=1}
[junit4:junit4] 2> 673639 T2224 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 673641 T2224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 673642 T2224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 673643 T2224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 673643 T2224 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 673644 T2224 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 673644 T2224 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 673645 T2224 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 673646 T2224 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 673646 T2224 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 673650 T2224 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 673654 T2224 oass.SolrIndexSearcher.<init> Opening Searcher@46fc29ff main
[junit4:junit4] 2> 673655 T2224 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 673655 T2224 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 673662 T2225 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@46fc29ff main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 673665 T2224 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 673665 T2224 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53322/sr collection:collection1 shard:shard1
[junit4:junit4] 2> 673670 T2224 oasc.ZkController.register We are http://127.0.0.1:53322/sr/collection1/ and leader is http://127.0.0.1:39944/sr/collection1/
[junit4:junit4] 2> 673670 T2224 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53322/sr
[junit4:junit4] 2> 673671 T2224 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 673671 T2224 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C2456 name=collection1 org.apache.solr.core.SolrCore@5d541d9e url=http://127.0.0.1:53322/sr/collection1 node=127.0.0.1:53322_sr C2456_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:53322_sr, base_url=http://127.0.0.1:53322/sr}
[junit4:junit4] 2> 673672 T2226 C2456 P53322 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 673672 T2226 C2456 P53322 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 673673 T2224 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 673673 T2226 C2456 P53322 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 673673 T2226 C2456 P53322 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 673675 T2165 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 673675 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 673675 T2226 C2456 P53322 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 673676 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 673687 T2203 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 673756 T2165 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 673758 T2165 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50010
[junit4:junit4] 2> 673758 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 673759 T2165 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 673759 T2165 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667
[junit4:junit4] 2> 673759 T2165 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667/solr.xml
[junit4:junit4] 2> 673759 T2165 oasc.CoreContainer.<init> New CoreContainer 510334839
[junit4:junit4] 2> 673760 T2165 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667/'
[junit4:junit4] 2> 673760 T2165 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667/'
[junit4:junit4] 2> 673830 T2165 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 673831 T2165 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 673831 T2165 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 673831 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 673831 T2165 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 673832 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 673832 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 673832 T2165 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 673832 T2165 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 673833 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 673836 T2165 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 673836 T2165 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42941/solr
[junit4:junit4] 2> 673836 T2165 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 673837 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 673839 T2238 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6adb7cc3 name:ZooKeeperConnection Watcher:127.0.0.1:42941 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 673839 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 673841 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 673844 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 673846 T2240 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58130412 name:ZooKeeperConnection Watcher:127.0.0.1:42941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 673846 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 673849 T2165 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 673944 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 673946 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53322_sr",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53322/sr"}
[junit4:junit4] 2> 673952 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 673952 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 673953 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 673952 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 673952 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 674688 T2203 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 674688 T2203 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node2&state=recovering&nodeName=127.0.0.1:53322_sr&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 674852 T2165 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50010_sr
[junit4:junit4] 2> 674854 T2165 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50010_sr
[junit4:junit4] 2> 674857 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 674857 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 674857 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 674858 T2194 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 674858 T2240 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 674858 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 674859 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 674860 T2223 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 674861 T2207 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 674861 T2187 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 674867 T2241 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 674868 T2241 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 674869 T2241 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 675460 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 675462 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50010_sr",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50010/sr"}
[junit4:junit4] 2> 675462 T2188 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 675462 T2188 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 675468 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 675469 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 675469 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 675468 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 675469 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 675870 T2241 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 675871 T2241 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667/collection1
[junit4:junit4] 2> 675871 T2241 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 675872 T2241 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 675872 T2241 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 675874 T2241 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667/collection1/'
[junit4:junit4] 2> 675875 T2241 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667/collection1/lib/README' to classloader
[junit4:junit4] 2> 675876 T2241 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 675904 T2241 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 675952 T2241 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 675953 T2241 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 675958 T2241 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 676370 T2241 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 676375 T2241 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 676378 T2241 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 676388 T2241 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 676391 T2241 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 676395 T2241 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 676396 T2241 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 676397 T2241 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 676397 T2241 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 676398 T2241 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 676398 T2241 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 676399 T2241 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 676399 T2241 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty3/
[junit4:junit4] 2> 676399 T2241 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b71c166
[junit4:junit4] 2> 676400 T2241 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty3
[junit4:junit4] 2> 676400 T2241 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty3/index/
[junit4:junit4] 2> 676401 T2241 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 676401 T2241 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty3/index
[junit4:junit4] 2> 676404 T2241 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d73c5d3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7c39811e),segFN=segments_1,generation=1}
[junit4:junit4] 2> 676404 T2241 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 676407 T2241 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 676407 T2241 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 676408 T2241 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 676409 T2241 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 676410 T2241 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 676411 T2241 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 676411 T2241 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 676412 T2241 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 676413 T2241 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 676418 T2241 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 676421 T2241 oass.SolrIndexSearcher.<init> Opening Searcher@4379d712 main
[junit4:junit4] 2> 676421 T2241 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 676422 T2241 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 676425 T2242 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4379d712 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 676426 T2241 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 676426 T2241 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50010/sr collection:collection1 shard:shard1
[junit4:junit4] 2> 676430 T2241 oasc.ZkController.register We are http://127.0.0.1:50010/sr/collection1/ and leader is http://127.0.0.1:39944/sr/collection1/
[junit4:junit4] 2> 676430 T2241 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50010/sr
[junit4:junit4] 2> 676430 T2241 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 676430 T2241 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C2457 name=collection1 org.apache.solr.core.SolrCore@256efb38 url=http://127.0.0.1:50010/sr/collection1 node=127.0.0.1:50010_sr C2457_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:50010_sr, base_url=http://127.0.0.1:50010/sr}
[junit4:junit4] 2> 676431 T2243 C2457 P50010 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 676431 T2241 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 676431 T2243 C2457 P50010 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 676432 T2243 C2457 P50010 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 676432 T2243 C2457 P50010 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 676433 T2165 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 676434 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 676433 T2243 C2457 P50010 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 676434 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 676439 T2202 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 676528 T2165 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 676530 T2165 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39475
[junit4:junit4] 2> 676531 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 676531 T2165 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 676531 T2165 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420
[junit4:junit4] 2> 676532 T2165 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420/solr.xml
[junit4:junit4] 2> 676532 T2165 oasc.CoreContainer.<init> New CoreContainer 876879168
[junit4:junit4] 2> 676532 T2165 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420/'
[junit4:junit4] 2> 676532 T2165 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420/'
[junit4:junit4] 2> 676592 T2165 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 676592 T2165 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 676592 T2165 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 676593 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 676593 T2165 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 676593 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 676593 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 676594 T2165 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 676594 T2165 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 676594 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 676598 T2165 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 676599 T2165 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42941/solr
[junit4:junit4] 2> 676599 T2165 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 676600 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 676601 T2255 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@781a5d46 name:ZooKeeperConnection Watcher:127.0.0.1:42941 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 676602 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 676603 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 676606 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 676607 T2257 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e78a7d name:ZooKeeperConnection Watcher:127.0.0.1:42941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 676607 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 676610 T2165 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C2458 name=collection1 org.apache.solr.core.SolrCore@5d541d9e url=http://127.0.0.1:53322/sr/collection1 node=127.0.0.1:53322_sr C2458_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:53322_sr, base_url=http://127.0.0.1:53322/sr}
[junit4:junit4] 2> 676690 T2226 C2458 P53322 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:39944/sr/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 676690 T2226 C2458 P53322 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:53322/sr START replicas=[http://127.0.0.1:39944/sr/collection1/] nUpdates=100
[junit4:junit4] 2> 676690 T2226 C2458 P53322 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 676691 T2226 C2458 P53322 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 676691 T2226 C2458 P53322 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 676691 T2226 C2458 P53322 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 676691 T2226 C2458 P53322 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 676691 T2226 C2458 P53322 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:39944/sr/collection1/. core=collection1
[junit4:junit4] 2> 676691 T2226 C2458 P53322 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C2459 name=collection1 org.apache.solr.core.SolrCore@35d6ca2b url=http://127.0.0.1:39944/sr/collection1 node=127.0.0.1:39944_sr C2459_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:39944_sr, base_url=http://127.0.0.1:39944/sr, leader=true}
[junit4:junit4] 2> 676697 T2199 C2459 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 676700 T2201 C2459 P39944 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 676702 T2201 C2459 P39944 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@673bbff8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2ee94c06),segFN=segments_1,generation=1}
[junit4:junit4] 2> 676703 T2201 C2459 P39944 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 676703 T2201 C2459 P39944 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 676703 T2201 C2459 P39944 oass.SolrIndexSearcher.<init> Opening Searcher@6aa257e5 realtime
[junit4:junit4] 2> 676703 T2201 C2459 P39944 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 676704 T2201 C2459 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4] 2> 676705 T2226 C2458 P53322 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 676705 T2226 C2458 P53322 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 676706 T2203 C2459 P39944 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 676706 T2203 C2459 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 676707 T2226 C2458 P53322 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 676707 T2226 C2458 P53322 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 676707 T2226 C2458 P53322 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 676707 T2226 C2458 P53322 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 676708 T2226 C2458 P53322 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 676976 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 676977 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50010_sr",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50010/sr"}
[junit4:junit4] 2> 676982 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53322_sr",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53322/sr"}
[junit4:junit4] 2> 676990 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 676990 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 676990 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 676990 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 676990 T2257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 676990 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 677439 T2202 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 677440 T2202 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node3&state=recovering&nodeName=127.0.0.1:50010_sr&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 677613 T2165 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39475_sr
[junit4:junit4] 2> 677615 T2165 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39475_sr
[junit4:junit4] 2> 677618 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 677619 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 677619 T2240 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 677618 T2257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 677618 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 677619 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 677619 T2194 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 677621 T2223 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 677621 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 677622 T2187 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 677623 T2257 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 677623 T2207 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 677630 T2259 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 677630 T2259 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 677633 T2259 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 678500 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 678501 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39475_sr",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39475/sr"}
[junit4:junit4] 2> 678501 T2188 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 678502 T2188 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 678508 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 678508 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 678508 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 678508 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 678508 T2257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 678508 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 678634 T2259 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 678634 T2259 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420/collection1
[junit4:junit4] 2> 678634 T2259 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 678635 T2259 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 678636 T2259 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 678638 T2259 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420/collection1/'
[junit4:junit4] 2> 678639 T2259 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420/collection1/lib/README' to classloader
[junit4:junit4] 2> 678639 T2259 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 678682 T2259 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 678711 T2259 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 678713 T2259 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 678716 T2259 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 679098 T2259 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 679103 T2259 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 679105 T2259 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 679123 T2259 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 679128 T2259 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 679133 T2259 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 679135 T2259 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 679135 T2259 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 679135 T2259 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 679137 T2259 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 679138 T2259 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 679138 T2259 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 679138 T2259 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty4/
[junit4:junit4] 2> 679139 T2259 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b71c166
[junit4:junit4] 2> 679140 T2259 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty4
[junit4:junit4] 2> 679140 T2259 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty4/index/
[junit4:junit4] 2> 679141 T2259 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 679141 T2259 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty4/index
[junit4:junit4] 2> 679143 T2259 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@71b09f32 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c5224d1),segFN=segments_1,generation=1}
[junit4:junit4] 2> 679144 T2259 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 679146 T2259 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 679147 T2259 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 679148 T2259 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 679149 T2259 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 679150 T2259 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 679150 T2259 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 679150 T2259 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 679151 T2259 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 679152 T2259 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 679156 T2259 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 679160 T2259 oass.SolrIndexSearcher.<init> Opening Searcher@10aee96d main
[junit4:junit4] 2> 679160 T2259 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 679160 T2259 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 679164 T2260 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@10aee96d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 679165 T2259 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 679165 T2259 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39475/sr collection:collection1 shard:shard1
[junit4:junit4] 2> 679169 T2259 oasc.ZkController.register We are http://127.0.0.1:39475/sr/collection1/ and leader is http://127.0.0.1:39944/sr/collection1/
[junit4:junit4] 2> 679169 T2259 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39475/sr
[junit4:junit4] 2> 679169 T2259 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 679169 T2259 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C2460 name=collection1 org.apache.solr.core.SolrCore@7056e8b4 url=http://127.0.0.1:39475/sr/collection1 node=127.0.0.1:39475_sr C2460_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:39475_sr, base_url=http://127.0.0.1:39475/sr}
[junit4:junit4] 2> 679170 T2261 C2460 P39475 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 679171 T2261 C2460 P39475 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 679171 T2259 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 679171 T2261 C2460 P39475 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 679172 T2261 C2460 P39475 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 679173 T2165 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 679173 T2261 C2460 P39475 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 679173 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 679174 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 679181 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 679182 T2202 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 679183 T2165 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30 for each attempt
[junit4:junit4] 2> 679183 T2165 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
[junit4:junit4] 2> 679184 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C2461 name=collection1 org.apache.solr.core.SolrCore@256efb38 url=http://127.0.0.1:50010/sr/collection1 node=127.0.0.1:50010_sr C2461_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:50010_sr, base_url=http://127.0.0.1:50010/sr}
[junit4:junit4] 2> 679441 T2243 C2461 P50010 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:39944/sr/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 679441 T2243 C2461 P50010 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50010/sr START replicas=[http://127.0.0.1:39944/sr/collection1/] nUpdates=100
[junit4:junit4] 2> 679442 T2243 C2461 P50010 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 679443 T2243 C2461 P50010 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 679444 T2243 C2461 P50010 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 679444 T2243 C2461 P50010 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 679444 T2243 C2461 P50010 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 679444 T2243 C2461 P50010 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:39944/sr/collection1/. core=collection1
[junit4:junit4] 2> 679445 T2243 C2461 P50010 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C2462 name=collection1 org.apache.solr.core.SolrCore@35d6ca2b url=http://127.0.0.1:39944/sr/collection1 node=127.0.0.1:39944_sr C2462_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:39944_sr, base_url=http://127.0.0.1:39944/sr, leader=true}
[junit4:junit4] 2> 679445 T2203 C2462 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 679451 T2201 C2462 P39944 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 679452 T2201 C2462 P39944 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 679452 T2201 C2462 P39944 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 679452 T2201 C2462 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 1
[junit4:junit4] 2> 679454 T2243 C2461 P50010 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 679454 T2243 C2461 P50010 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 679456 T2203 C2462 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 679457 T2243 C2461 P50010 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 679457 T2243 C2461 P50010 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 679457 T2243 C2461 P50010 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 679458 T2243 C2461 P50010 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 679459 T2243 C2461 P50010 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 680014 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 680015 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39475_sr",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39475/sr"}
[junit4:junit4] 2> 680019 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50010_sr",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50010/sr"}
[junit4:junit4] 2> 680022 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 680022 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 680022 T2257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 680022 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 680022 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 680022 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 680182 T2202 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 680183 T2202 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node4&state=recovering&nodeName=127.0.0.1:39475_sr&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 680186 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 681189 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C2460_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:39475_sr, base_url=http://127.0.0.1:39475/sr}
[junit4:junit4] 2> 682184 T2261 C2460 P39475 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:39944/sr/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 682185 T2261 C2460 P39475 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39475/sr START replicas=[http://127.0.0.1:39944/sr/collection1/] nUpdates=100
[junit4:junit4] 2> 682185 T2261 C2460 P39475 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 682186 T2261 C2460 P39475 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 682186 T2261 C2460 P39475 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 682187 T2261 C2460 P39475 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 682187 T2261 C2460 P39475 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 682187 T2261 C2460 P39475 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:39944/sr/collection1/. core=collection1
[junit4:junit4] 2> 682188 T2261 C2460 P39475 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 682188 T2203 C2462 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 682191 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 682195 T2201 C2462 P39944 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 682195 T2201 C2462 P39944 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 682196 T2201 C2462 P39944 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 682196 T2201 C2462 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 1
[junit4:junit4] 2> 682197 T2261 C2460 P39475 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 682197 T2261 C2460 P39475 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 682200 T2202 C2462 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 682201 T2261 C2460 P39475 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 682201 T2261 C2460 P39475 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 682201 T2261 C2460 P39475 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 682201 T2261 C2460 P39475 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 682203 T2261 C2460 P39475 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 683032 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 683034 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39475_sr",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39475/sr"}
[junit4:junit4] 2> 683041 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 683041 T2257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 683041 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 683041 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 683041 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 683041 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 683194 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 683196 T2165 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C2463 name=collection1 org.apache.solr.core.SolrCore@547e62a7 url=http://127.0.0.1:40343/sr/collection1 node=127.0.0.1:40343_sr C2463_STATE=coll:control_collection core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:40343_sr, base_url=http://127.0.0.1:40343/sr, leader=true}
[junit4:junit4] 2> 683206 T2180 C2463 P40343 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 683209 T2180 C2463 P40343 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@599cdbaf lockFactory=org.apache.lucene.store.NativeFSLockFactory@475f075f),segFN=segments_1,generation=1}
[junit4:junit4] 2> 683210 T2180 C2463 P40343 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 683211 T2180 C2463 P40343 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 683212 T2180 C2463 P40343 oass.SolrIndexSearcher.<init> Opening Searcher@6cdd19d7 main
[junit4:junit4] 2> 683212 T2180 C2463 P40343 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 683213 T2191 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6cdd19d7 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 683214 T2180 C2463 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> C2461_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:50010_sr, base_url=http://127.0.0.1:50010/sr}
[junit4:junit4] 2> 683227 T2236 C2461 P50010 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:39944/sr/collection1/, StdNode: http://127.0.0.1:53322/sr/collection1/, StdNode: http://127.0.0.1:50010/sr/collection1/, StdNode: http://127.0.0.1:39475/sr/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 683232 T2203 C2462 P39944 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 683233 T2203 C2462 P39944 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 683234 T2203 C2462 P39944 oass.SolrIndexSearcher.<init> Opening Searcher@7654ebb1 main
[junit4:junit4] 2> 683234 T2203 C2462 P39944 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 683235 T2209 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7654ebb1 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 683237 T2234 C2461 P50010 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 683237 T2203 C2462 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 683240 T2234 C2461 P50010 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d73c5d3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7c39811e),segFN=segments_1,generation=1}
[junit4:junit4] 2> 683241 T2234 C2461 P50010 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 683242 T2234 C2461 P50010 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> ASYNC NEW_CORE C2464 name=collection1 org.apache.solr.core.SolrCore@5d541d9e url=http://127.0.0.1:53322/sr/collection1 node=127.0.0.1:53322_sr C2464_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:53322_sr, base_url=http://127.0.0.1:53322/sr}
[junit4:junit4] 2> 683242 T2217 C2464 P53322 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 683243 T2234 C2461 P50010 oass.SolrIndexSearcher.<init> Opening Searcher@1b3ecbfb main
[junit4:junit4] 2> 683244 T2234 C2461 P50010 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> C2460_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:39475_sr, base_url=http://127.0.0.1:39475/sr}
[junit4:junit4] 2> 683246 T2250 C2460 P39475 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 683246 T2217 C2464 P53322 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60317470 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48e612de),segFN=segments_1,generation=1}
[junit4:junit4] 2> 683246 T2242 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1b3ecbfb main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 683247 T2217 C2464 P53322 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 683248 T2234 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4] 2> 683248 T2217 C2464 P53322 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 683248 T2250 C2460 P39475 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@71b09f32 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c5224d1),segFN=segments_1,generation=1}
[junit4:junit4] 2> 683250 T2217 C2464 P53322 oass.SolrIndexSearcher.<init> Opening Searcher@6dcdfe8 main
[junit4:junit4] 2> 683251 T2250 C2460 P39475 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 683251 T2217 C2464 P53322 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 683252 T2250 C2460 P39475 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 683252 T2225 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6dcdfe8 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 683253 T2250 C2460 P39475 oass.SolrIndexSearcher.<init> Opening Searcher@5f488d27 main
[junit4:junit4] 2> 683253 T2217 C2464 P53322 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4] 2> 683253 T2250 C2460 P39475 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 683255 T2260 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5f488d27 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 683255 T2250 C2460 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 10
[junit4:junit4] 2> 683256 T2236 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 29
[junit4:junit4] 2> 683258 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 683261 T2201 C2462 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 683264 T2218 C2464 P53322 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 683267 T2235 C2461 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 683269 T2251 C2460 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 685275 T2183 C2463 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1439069496415354880)} 0 2
[junit4:junit4] 2> 685285 T2250 C2460 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={update.distrib=FROMLEADER&_version_=-1439069496422694912&update.from=http://127.0.0.1:39944/sr/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439069496422694912)} 0 1
[junit4:junit4] 2> 685286 T2217 C2464 P53322 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={update.distrib=FROMLEADER&_version_=-1439069496422694912&update.from=http://127.0.0.1:39944/sr/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439069496422694912)} 0 2
[junit4:junit4] 2> 685286 T2233 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={update.distrib=FROMLEADER&_version_=-1439069496422694912&update.from=http://127.0.0.1:39944/sr/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439069496422694912)} 0 2
[junit4:junit4] 2> 685287 T2203 C2462 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1439069496422694912)} 0 7
[junit4:junit4] 2> 685288 T2236 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 11
[junit4:junit4] 2> 685294 T2181 C2463 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {add=[0 (1439069496434229248)]} 0 3
[junit4:junit4] 2> 685305 T2216 C2464 P53322 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439069496442617856)]} 0 2
[junit4:junit4] 2> 685305 T2234 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439069496442617856)]} 0 2
[junit4:junit4] 2> 685305 T2251 C2460 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439069496442617856)]} 0 2
[junit4:junit4] 2> 685307 T2199 C2462 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:53322/sr/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1439069496442617856)]} 0 8
[junit4:junit4] 2> 685308 T2219 C2464 P53322 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {add=[0]} 0 12
[junit4:junit4] 2> 685312 T2182 C2463 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {add=[1 (1439069496455200768)]} 0 1
[junit4:junit4] 2> 685323 T2233 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439069496462540800)]} 0 2
[junit4:junit4] 2> 685323 T2252 C2460 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439069496462540800)]} 0 2
[junit4:junit4] 2> 685323 T2217 C2464 P53322 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439069496462540800)]} 0 2
[junit4:junit4] 2> 685324 T2202 C2462 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:50010/sr/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1439069496462540800)]} 0 7
[junit4:junit4] 2> 685325 T2235 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {add=[1]} 0 10
[junit4:junit4] 2> 685329 T2180 C2463 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {add=[2 (1439069496473026560)]} 0 1
[junit4:junit4] 2> 685335 T2201 C2462 P39944 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:53322/sr/collection1/]
[junit4:junit4] 2> 685336 T2201 C2462 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:53322/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:true
[junit4:junit4] 2> 685336 T2201 C2462 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:50010/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:false
[junit4:junit4] 2> 685337 T2201 C2462 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:39475/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:false
[junit4:junit4] 2> 685342 T2236 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[2 (1439069496482463744)]} 0 1
[junit4:junit4] 2> 685342 T2253 C2460 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[2 (1439069496482463744)]} 0 1
[junit4:junit4] 2> 685344 T2201 C2462 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:53322/sr/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[2 (1439069496482463744)]} 0 9
[junit4:junit4] 2> 685345 T2218 C2464 P53322 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[2]} 0 13
[junit4:junit4] 2> 685349 T2183 C2463 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {add=[3 (1439069496493998080)]} 0 2
[junit4:junit4] 2> 685354 T2203 C2462 P39944 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:53322/sr/collection1/]
[junit4:junit4] 2> 685355 T2203 C2462 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:53322/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:true
[junit4:junit4] 2> 685355 T2203 C2462 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:50010/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:false
[junit4:junit4] 2> 685356 T2203 C2462 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:39475/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:false
[junit4:junit4] 2> 685361 T2233 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[3 (1439069496502386688)]} 0 2
[junit4:junit4] 2> 685361 T2250 C2460 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[3 (1439069496502386688)]} 0 2
[junit4:junit4] 2> 685362 T2203 C2462 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:50010/sr/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[3 (1439069496502386688)]} 0 8
[junit4:junit4] 2> 685363 T2234 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&test.distrib.skip.servers=http://127.0.0.1:50010/sr/collection1/&wt=javabin&version=2} {add=[3]} 0 11
[junit4:junit4] 2> 685365 T2181 C2463 P40343 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 685369 T2181 C2463 P40343 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@599cdbaf lockFactory=org.apache.lucene.store.NativeFSLockFactory@475f075f),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@599cdbaf lockFactory=org.apache.lucene.store.NativeFSLockFactory@475f075f),segFN=segments_2,generation=2}
[junit4:junit4] 2> 685370 T2181 C2463 P40343 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 685372 T2181 C2463 P40343 oass.SolrIndexSearcher.<init> Opening Searcher@6e7780f0 main
[junit4:junit4] 2> 685373 T2181 C2463 P40343 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 685374 T2191 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6e7780f0 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 685374 T2181 C2463 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4] 2> 685377 T2199 C2462 P39944 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:39944/sr/collection1/, StdNode: http://127.0.0.1:53322/sr/collection1/, StdNode: http://127.0.0.1:50010/sr/collection1/, StdNode: http://127.0.0.1:39475/sr/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 685379 T2202 C2462 P39944 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 685380 T2251 C2460 P39475 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 685379 T2235 C2461 P50010 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 685379 T2216 C2464 P53322 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 685385 T2251 C2460 P39475 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@71b09f32 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c5224d1),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@71b09f32 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c5224d1),segFN=segments_2,generation=2}
[junit4:junit4] 2> ASYNC NEW_CORE C2465 name=collection1 org.apache.solr.core.SolrCore@7056e8b4 url=http://127.0.0.1:39475/sr/collection1 node=127.0.0.1:39475_sr C2465_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:39475_sr, base_url=http://127.0.0.1:39475/sr}
[junit4:junit4] 2> 685405 T2251 C2465 P39475 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> ASYNC NEW_CORE C2466 name=collection1 org.apache.solr.core.SolrCore@256efb38 url=http://127.0.0.1:50010/sr/collection1 node=127.0.0.1:50010_sr C2466_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:50010_sr, base_url=http://127.0.0.1:50010/sr}
[junit4:junit4] 2> 685406 T2235 C2466 P50010 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d73c5d3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7c39811e),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d73c5d3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7c39811e),segFN=segments_2,generation=2}
[junit4:junit4] 2> ASYNC NEW_CORE C2467 name=collection1 org.apache.solr.core.SolrCore@35d6ca2b url=http://127.0.0.1:39944/sr/collection1 node=127.0.0.1:39944_sr C2467_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:39944_sr, base_url=http://127.0.0.1:39944/sr, leader=true}
[junit4:junit4] 2> 685406 T2202 C2467 P39944 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@673bbff8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2ee94c06),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@673bbff8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2ee94c06),segFN=segments_2,generation=2}
[junit4:junit4] 2> 685407 T2251 C2465 P39475 oass.SolrIndexSearcher.<init> Opening Searcher@29b5bc49 main
[junit4:junit4] 2> 685407 T2202 C2467 P39944 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 685407 T2235 C2466 P50010 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> ASYNC NEW_CORE C2468 name=collection1 org.apache.solr.core.SolrCore@5d541d9e url=http://127.0.0.1:53322/sr/collection1 node=127.0.0.1:53322_sr C2468_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:53322_sr, base_url=http://127.0.0.1:53322/sr}
[junit4:junit4] 2> 685406 T2216 C2468 P53322 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60317470 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48e612de),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60317470 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48e612de),segFN=segments_2,generation=2}
[junit4:junit4] 2> 685408 T2216 C2468 P53322 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 685408 T2260 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@29b5bc49 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 685407 T2251 C2465 P39475 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 685409 T2251 C2465 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 29
[junit4:junit4] 2> 685408 T2235 C2466 P50010 oass.SolrIndexSearcher.<init> Opening Searcher@3fb589b1 main
[junit4:junit4] 2> 685408 T2202 C2467 P39944 oass.SolrIndexSearcher.<init> Opening Searcher@34bc6922 main
[junit4:junit4] 2> 685411 T2235 C2466 P50010 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 685410 T2216 C2468 P53322 oass.SolrIndexSearcher.<init> Opening Searcher@67edd42c main
[junit4:junit4] 2> 685411 T2242 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3fb589b1 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 685411 T2202 C2467 P39944 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 685412 T2235 C2466 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 33
[junit4:junit4] 2> 685411 T2216 C2468 P53322 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 685411 T2209 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@34bc6922 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 685412 T2225 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@67edd42c main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C2)}
[junit4:junit4] 2> 685413 T2202 C2467 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 34
[junit4:junit4] 2> 685413 T2216 C2468 P53322 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 34
[junit4:junit4] 2> 685413 T2199 C2467 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 36
[junit4:junit4] 2> 685414 T2165 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):330
[junit4:junit4] 2> 685414 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 685415 T2165 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 685418 T2201 C2467 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 685420 T2219 C2468 P53322 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0
[junit4:junit4] 2> 685422 T2236 C2466 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 685425 T2252 C2465 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 685425 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 685431 T2233 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4] 2> 685431 T2233 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 685437 T2203 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4] 2> 685438 T2203 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39944/sr/collection1/
[junit4:junit4] 2> 685438 T2203 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39944/sr START replicas=[http://127.0.0.1:53322/sr/collection1/, http://127.0.0.1:50010/sr/collection1/, http://127.0.0.1:39475/sr/collection1/] nUpdates=100
[junit4:junit4] 2> 685440 T2217 C2468 P53322 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 685440 T2250 C2465 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 685440 T2234 C2466 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 685441 T2203 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39944/sr Received 3 versions from 127.0.0.1:53322/sr/collection1/
[junit4:junit4] 2> 685442 T2203 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39944/sr Our versions are newer. ourLowThreshold=1439069496422694912 otherHigh=1439069496462540800
[junit4:junit4] 2> 685442 T2203 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39944/sr Received 5 versions from 127.0.0.1:39475/sr/collection1/
[junit4:junit4] 2> 685443 T2203 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39944/sr Our versions are newer. ourLowThreshold=1439069496422694912 otherHigh=1439069496482463744
[junit4:junit4] 2> 685443 T2203 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39944/sr Received 5 versions from 127.0.0.1:50010/sr/collection1/
[junit4:junit4] 2> 685443 T2203 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39944/sr Our versions are newer. ourLowThreshold=1439069496422694912 otherHigh=1439069496482463744
[junit4:junit4] 2> 685444 T2203 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39944/sr DONE. sync succeeded
[junit4:junit4] 2> 685444 T2203 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 685445 T2203 oasc.SyncStrategy.syncToMe http://127.0.0.1:39944/sr/collection1/: try and ask http://127.0.0.1:53322/sr/collection1/ to sync
[junit4:junit4] 2> 685445 T2203 oasc.SyncStrategy.syncToMe http://127.0.0.1:39944/sr/collection1/: try and ask http://127.0.0.1:50010/sr/collection1/ to sync
[junit4:junit4] 2> 685446 T2203 oasc.SyncStrategy.syncToMe http://127.0.0.1:39944/sr/collection1/: try and ask http://127.0.0.1:39475/sr/collection1/ to sync
[junit4:junit4] 2> 685447 T2218 C2468 P53322 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:53322/sr START replicas=[http://127.0.0.1:39944/sr/collection1/] nUpdates=100
[junit4:junit4] 2> 685447 T2235 C2466 P50010 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50010/sr START replicas=[http://127.0.0.1:39944/sr/collection1/] nUpdates=100
[junit4:junit4] 2> 685448 T2253 C2465 P39475 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39475/sr START replicas=[http://127.0.0.1:39944/sr/collection1/] nUpdates=100
[junit4:junit4] 2> 685449 T2199 C2467 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 685449 T2202 C2467 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 685450 T2218 C2468 P53322 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:53322/sr Received 5 versions from 127.0.0.1:39944/sr/collection1/
[junit4:junit4] 2> 685450 T2201 C2467 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 685450 T2235 C2466 P50010 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:50010/sr Received 5 versions from 127.0.0.1:39944/sr/collection1/
[junit4:junit4] 2> 685451 T2253 C2465 P39475 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39475/sr Received 5 versions from 127.0.0.1:39944/sr/collection1/
[junit4:junit4] 2> 685452 T2253 C2465 P39475 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39475/sr Our versions are newer. ourLowThreshold=1439069496422694912 otherHigh=1439069496482463744
[junit4:junit4] 2> 685451 T2218 C2468 P53322 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:53322/sr Requesting updates from 127.0.0.1:39944/sr/collection1/n=2 versions=[1439069496502386688, 1439069496482463744]
[junit4:junit4] 2> 685452 T2253 C2465 P39475 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39475/sr DONE. sync succeeded
[junit4:junit4] 2> 685453 T2253 C2465 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={sync=http://127.0.0.1:39944/sr/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=5
[junit4:junit4] 2> 685451 T2235 C2466 P50010 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:50010/sr Our versions are newer. ourLowThreshold=1439069496422694912 otherHigh=1439069496482463744
[junit4:junit4] 2> 685454 T2203 oasc.SyncStrategy.syncToMe http://127.0.0.1:39944/sr/collection1/: sync completed with http://127.0.0.1:39475/sr/collection1/
[junit4:junit4] 2> 685454 T2235 C2466 P50010 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50010/sr DONE. sync succeeded
[junit4:junit4] 2> 685455 T2199 C2467 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={distrib=false&getUpdates=1439069496502386688,1439069496482463744&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 685455 T2235 C2466 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={sync=http://127.0.0.1:39944/sr/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=8
[junit4:junit4] 2> 685456 T2203 oasc.SyncStrategy.syncToMe http://127.0.0.1:39944/sr/collection1/: sync completed with http://127.0.0.1:50010/sr/collection1/
[junit4:junit4] 2> 685459 T2218 C2468 P53322 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1439069496482463744), 3 (1439069496502386688)]} 0 3
[junit4:junit4] 2> 685459 T2218 C2468 P53322 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:53322/sr DONE. sync succeeded
[junit4:junit4] 2> 685460 T2218 C2468 P53322 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={sync=http://127.0.0.1:39944/sr/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=14
[junit4:junit4] 2> 685461 T2203 oasc.SyncStrategy.syncToMe http://127.0.0.1:39944/sr/collection1/: sync completed with http://127.0.0.1:53322/sr/collection1/
[junit4:junit4] 2> 685461 T2203 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=24
[junit4:junit4] 2> 685462 T2233 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=31
[junit4:junit4] 2> 685463 T2165 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 685464 T2165 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 685465 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 685466 T2165 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C2469 name=collection1 org.apache.solr.core.SolrCore@547e62a7 url=http://127.0.0.1:40343/sr/collection1 node=127.0.0.1:40343_sr C2469_STATE=coll:control_collection core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:40343_sr, base_url=http://127.0.0.1:40343/sr, leader=true}
[junit4:junit4] 2> 685469 T2182 C2469 P40343 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 685470 T2182 C2469 P40343 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 685470 T2182 C2469 P40343 oass.SolrIndexSearcher.<init> Opening Searcher@1dfbe8df main
[junit4:junit4] 2> 685471 T2182 C2469 P40343 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 685472 T2191 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1dfbe8df main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 685472 T2182 C2469 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4] 2> 685475 T2252 C2465 P39475 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:39944/sr/collection1/, StdNode: http://127.0.0.1:53322/sr/collection1/, StdNode: http://127.0.0.1:50010/sr/collection1/, StdNode: http://127.0.0.1:39475/sr/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 685479 T2236 C2466 P50010 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 685479 T2250 C2465 P39475 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 685480 T2236 C2466 P50010 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 685479 T2202 C2467 P39944 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 685479 T2216 C2468 P53322 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 685482 T2202 C2467 P39944 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 685481 T2236 C2466 P50010 oass.SolrIndexSearcher.<init> Opening Searcher@4488d4ca main
[junit4:junit4] 2> 685481 T2250 C2465 P39475 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 685483 T2236 C2466 P50010 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 685483 T2202 C2467 P39944 oass.SolrIndexSearcher.<init> Opening Searcher@7a992c6e main
[junit4:junit4] 2> 685484 T2250 C2465 P39475 oass.SolrIndexSearcher.<init> Opening Searcher@3aea2fdd main
[junit4:junit4] 2> 685485 T2242 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4488d4ca main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 685485 T2202 C2467 P39944 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 685486 T2209 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7a992c6e main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 685486 T2260 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3aea2fdd main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 685485 T2216 C2468 P53322 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60317470 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48e612de),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60317470 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48e612de),segFN=segments_3,generation=3}
[junit4:junit4] 2> 685487 T2216 C2468 P53322 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 685485 T2250 C2465 P39475 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 685487 T2202 C2467 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 685486 T2236 C2466 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 2> 685489 T2216 C2468 P53322 oass.SolrIndexSearcher.<init> Opening Searcher@2fe11990 main
[junit4:junit4] 2> 685488 T2250 C2465 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4] 2> 685490 T2216 C2468 P53322 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 685491 T2225 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2fe11990 main{StandardDirectoryReader(segments_3:5:nrt _0(5.0):C2 _1(5.0):C2)}
[junit4:junit4] 2> 685492 T2216 C2468 P53322 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 13
[junit4:junit4] 2> 685493 T2252 C2465 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 18
[junit4:junit4] 2> 685494 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 685498 T2201 C2467 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 685501 T2219 C2468 P53322 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 685504 T2234 C2466 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 685506 T2251 C2465 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 687508 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> check const of shard1
[junit4:junit4] 2> client0
[junit4:junit4] 2> PROPS:core_node1:{"state":"active","core":"collection1","node_name":"127.0.0.1:39944_sr","base_url":"http://127.0.0.1:39944/sr","leader":"true"}
[junit4:junit4] 2> 687512 T2202 C2467 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client1
[junit4:junit4] 2> PROPS:core_node2:{"state":"active","core":"collection1","node_name":"127.0.0.1:53322_sr","base_url":"http://127.0.0.1:53322/sr"}
[junit4:junit4] 2> 687515 T2216 C2468 P53322 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client2
[junit4:junit4] 2> PROPS:core_node3:{"state":"active","core":"collection1","node_name":"127.0.0.1:50010_sr","base_url":"http://127.0.0.1:50010/sr"}
[junit4:junit4] 2> 687518 T2236 C2466 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client3
[junit4:junit4] 2> PROPS:core_node4:{"state":"active","core":"collection1","node_name":"127.0.0.1:39475_sr","base_url":"http://127.0.0.1:39475/sr"}
[junit4:junit4] 2> 687521 T2252 C2465 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> 687524 T2253 C2465 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={q=*:*&wt=javabin&version=2} hits=4 status=0 QTime=0
[junit4:junit4] 2> 687530 T2181 C2469 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {add=[4 (1439069498779893760)]} 0 2
[junit4:junit4] 2> 687536 T2199 C2467 P39944 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:53322/sr/collection1/]
[junit4:junit4] 2> 687537 T2199 C2467 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:53322/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:true
[junit4:junit4] 2> 687537 T2199 C2467 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:50010/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:false
[junit4:junit4] 2> 687537 T2199 C2467 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:39475/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:false
[junit4:junit4] 2> 687544 T2234 C2466 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[4 (1439069498790379520)]} 0 2
[junit4:junit4] 2> 687544 T2251 C2465 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[4 (1439069498790379520)]} 0 2
[junit4:junit4] 2> 687545 T2199 C2467 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39475/sr/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[4 (1439069498790379520)]} 0 9
[junit4:junit4] 2> 687546 T2250 C2465 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[4]} 0 13
[junit4:junit4] 2> 687546 T2165 oasc.ChaosMonkey.monkeyLog monkey: kill shard! 39944
[junit4:junit4] 2> 687569 T2165 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=528298614
[junit4:junit4] 2> 689055 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 689056 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:39944_sr",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39944/sr"}
[junit4:junit4] 2> 689057 T2188 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4:junit4] 2> 689064 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 689064 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 689064 T2257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 689064 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 689064 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 689064 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 690572 T2165 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 690573 T2165 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 690575 T2165 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@35d6ca2b
[junit4:junit4] 2> 690580 T2165 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=5,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 690582 T2165 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 690582 T2165 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 690583 T2165 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> C2467_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:39944_sr, base_url=http://127.0.0.1:39944/sr, leader=true}
[junit4:junit4] 2> 690586 T2165 C2467 P39944 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@673bbff8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2ee94c06),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@673bbff8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2ee94c06),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@673bbff8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2ee94c06),segFN=segments_3,generation=3}
[junit4:junit4] 2> 690587 T2165 C2467 P39944 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 690590 T2165 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 690591 T2165 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 690591 T2165 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1/index;done=false>>]
[junit4:junit4] 2> 690592 T2165 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1/index
[junit4:junit4] 2> 690592 T2165 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1;done=false>>]
[junit4:junit4] 2> 690593 T2165 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1
[junit4:junit4] 2> 690596 T2257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 690596 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 690596 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 690597 T2240 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 690597 T2207 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 690598 T2165 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/sr,null}
[junit4:junit4] 2> 690597 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 690598 T2194 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 690598 T2207 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 690600 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 690600 T2223 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4] 2> 690600 T2257 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 690600 T2187 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 690603 T2223 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 690603 T2223 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 690604 T2223 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 690604 T2223 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53322/sr/collection1/
[junit4:junit4] 2> 690604 T2223 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:53322/sr START replicas=[http://127.0.0.1:39944/sr/collection1/, http://127.0.0.1:50010/sr/collection1/, http://127.0.0.1:39475/sr/collection1/] nUpdates=100
[junit4:junit4] 2> 690607 T2223 oasu.PeerSync.handleResponse WARN PeerSync: core=collection1 url=http://127.0.0.1:53322/sr couldn't connect to http://127.0.0.1:39944/sr/collection1/, counting as success
[junit4:junit4] 2> 690608 T2234 C2466 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 690608 T2251 C2465 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 690609 T2223 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:53322/sr Received 6 versions from 127.0.0.1:50010/sr/collection1/
[junit4:junit4] 2> 690609 T2223 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:53322/sr Requesting updates from 127.0.0.1:50010/sr/collection1/n=1 versions=[1439069498790379520]
[junit4:junit4] 2> 690610 T2223 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:53322/sr Received 6 versions from 127.0.0.1:39475/sr/collection1/
[junit4:junit4] 2> 690610 T2223 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:53322/sr Our versions are newer. ourLowThreshold=1439069496422694912 otherHigh=1439069496502386688
[junit4:junit4] 2> 690612 T2233 C2466 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={distrib=false&getUpdates=1439069498790379520&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 690615 T2223 oasup.LogUpdateProcessor.finish [collection1] {add=[4 (1439069498790379520)]} 0 2
[junit4:junit4] 2> 690615 T2223 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:53322/sr DONE. sync succeeded
[junit4:junit4] 2> 690616 T2223 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 690616 T2223 oasc.SyncStrategy.syncToMe http://127.0.0.1:53322/sr/collection1/: try and ask http://127.0.0.1:39944/sr/collection1/ to sync
[junit4:junit4] 2> 690616 T2223 oasc.SyncStrategy.syncToMe http://127.0.0.1:53322/sr/collection1/: try and ask http://127.0.0.1:50010/sr/collection1/ to sync
[junit4:junit4] 2> 690617 T2223 oasc.SyncStrategy.syncToMe http://127.0.0.1:53322/sr/collection1/: try and ask http://127.0.0.1:39475/sr/collection1/ to sync
[junit4:junit4] 2> 690618 T2223 oasc.SolrException.log ERROR Sync request error: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:39944/sr/collection1
[junit4:junit4] 2> 690618 T2223 oasc.SyncStrategy.syncToMe http://127.0.0.1:53322/sr/collection1/: Sync failed - asking replica (http://127.0.0.1:39944/sr/collection1/) to recover.
[junit4:junit4] 2> 690619 T2294 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 690619 T2236 C2466 P50010 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50010/sr START replicas=[http://127.0.0.1:53322/sr/collection1/] nUpdates=100
[junit4:junit4] 2> 690620 T2253 C2465 P39475 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39475/sr START replicas=[http://127.0.0.1:53322/sr/collection1/] nUpdates=100
[junit4:junit4] 2> 690623 T2216 C2468 P53322 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 690623 T2219 C2468 P53322 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 690624 T2236 C2466 P50010 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:50010/sr Received 6 versions from 127.0.0.1:53322/sr/collection1/
[junit4:junit4] 2> 690624 T2253 C2465 P39475 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39475/sr Received 6 versions from 127.0.0.1:53322/sr/collection1/
[junit4:junit4] 2> 690624 T2236 C2466 P50010 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:50010/sr Our versions are newer. ourLowThreshold=1439069496442617856 otherHigh=1439069496502386688
[junit4:junit4] 2> 690625 T2236 C2466 P50010 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50010/sr DONE. sync succeeded
[junit4:junit4] 2> 690625 T2253 C2465 P39475 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39475/sr Our versions are newer. ourLowThreshold=1439069496442617856 otherHigh=1439069496502386688
[junit4:junit4] 2> 690626 T2236 C2466 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={sync=http://127.0.0.1:53322/sr/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=7
[junit4:junit4] 2> 690627 T2253 C2465 P39475 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39475/sr DONE. sync succeeded
[junit4:junit4] 2> 690627 T2223 oasc.SyncStrategy.syncToMe http://127.0.0.1:53322/sr/collection1/: sync completed with http://127.0.0.1:50010/sr/collection1/
[junit4:junit4] 2> 690628 T2253 C2465 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={sync=http://127.0.0.1:53322/sr/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=8
[junit4:junit4] 2> 690629 T2294 oasc.SolrException.log ERROR http://127.0.0.1:53322/sr/collection1/: Could not tell a replica to recover:org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:39944/sr
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4] 2> at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:297)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:39944 refused
[junit4:junit4] 2> at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
[junit4:junit4] 2> at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4:junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4:junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4:junit4] 2> ... 5 more
[junit4:junit4] 2> Caused by: java.net.ConnectException: Connection refused
[junit4:junit4] 2> at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4:junit4] 2> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4:junit4] 2> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4:junit4] 2> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4:junit4] 2> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[junit4:junit4] 2> at java.net.Socket.connect(Socket.java:579)
[junit4:junit4] 2> at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4:junit4] 2> at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4:junit4] 2> ... 12 more
[junit4:junit4] 2>
[junit4:junit4] 2> 690629 T2223 oasc.SyncStrategy.syncToMe http://127.0.0.1:53322/sr/collection1/: sync completed with http://127.0.0.1:39475/sr/collection1/
[junit4:junit4] 2> 690630 T2223 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53322/sr/collection1/ shard1
[junit4:junit4] 2> 690630 T2223 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 690636 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 690638 T2223 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 690650 T2165 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 39944
[junit4:junit4] 2> 692074 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 692075 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39944_sr",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39944/sr"}
[junit4:junit4] 2> 692090 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live
[...truncated too long message...]
:"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:39475_sr",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:39475/sr"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"core_node1":{
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:40343_sr",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:40343/sr",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:40343_sr (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:50010_sr (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:39475_sr (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:39944_sr (0)
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/collections (2)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89941842148524044-core_node4-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89941842148524046-core_node1-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89941842148524042-core_node3-n_0000000002 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (1)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:50010_sr",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:50010/sr"}
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89941842148524035-core_node1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:40343_sr",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:40343/sr"}
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/election (4)
[junit4:junit4] 1> /solr/overseer_elect/election/89941842148524046-127.0.0.1:39944_sr-n_0000000005 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89941842148524044-127.0.0.1:39475_sr-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89941842148524035-127.0.0.1:40343_sr-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89941842148524042-127.0.0.1:50010_sr-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89941842148524035-127.0.0.1:40343_sr-n_0000000000"}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=C103F4B8ABD96C0D -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_IE -Dtests.timezone=America/Cordoba -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 99.6s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:39944/sr/collection1lastClient and got 255 from http://127.0.0.1:50010/sr/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([C103F4B8ABD96C0D:40E57AA0DC860C31]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1045)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:237)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 763848 T2165 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 99606 T2164 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 763959 T2240 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 763959 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 763960 T2240 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=500), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=500)), sim=DefaultSimilarity, locale=en_IE, timezone=America/Cordoba
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=8,threads=3,free=137002440,total=312283136
[junit4:junit4] 2> NOTE: All tests run in this JVM: [PeerSyncTest, BasicFunctionalityTest, TestDynamicFieldResource, FastVectorHighlighterTest, TestFieldTypeResource, JsonLoaderTest, TestFunctionQuery, AnalysisAfterCoreReloadTest, TestLMDirichletSimilarityFactory, MinimalSchemaTest, DirectUpdateHandlerOptimizeTest, BasicDistributedZkTest, UUIDFieldTest, TestCopyFieldCollectionResource, TestSolrCoreProperties, TestTrie, OverseerCollectionProcessorTest, TestPhraseSuggestions, TestDefaultSearchFieldResource, TestDocSet, ClusterStateUpdateTest, LeaderElectionTest, RAMDirectoryFactoryTest, TestPseudoReturnFields, ShowFileRequestHandlerTest, TermVectorComponentTest, MoreLikeThisHandlerTest, DistributedQueryElevationComponentTest, TestLFUCache, TestManagedSchemaFieldResource, TestFastWriter, BadCopyFieldTest, TestSolrIndexConfig, DistributedTermsComponentTest, TestSolrDeletionPolicy2, SimpleFacetsTest, TestPluginEnable, TestValueSourceCache, DocValuesTest, TestStressRecovery, RecoveryZkTest, DirectSolrConnectionTest, TestQueryUtils, QueryElevationComponentTest, TestPropInjectDefaults, TestStressVersions, TestWriterPerf, SliceStateTest, TestGroupingSearch, TestFastLRUCache, SpellingQueryConverterTest, TestReversedWildcardFilterFactory, BlockDirectoryTest, BadComponentTest, CircularListTest, TestBadConfig, TestFieldTypeCollectionResource, IndexSchemaTest, UnloadDistributedZkTest, QueryParsingTest, TestPerFieldSimilarity, SpatialFilterTest, AliasIntegrationTest, TestSearchPerf, TestJoin, PreAnalyzedUpdateProcessorTest, SchemaVersionSpecificBehaviorTest, ParsingFieldUpdateProcessorsTest, SynonymTokenizerTest, TestSolrQueryParserDefaultOperatorResource, ScriptEngineTest, WordBreakSolrSpellCheckerTest, SuggesterFSTTest, TestQuerySenderListener, TestDocumentBuilder, BasicDistributedZk2Test, SyncSliceTest]
[junit4:junit4] Completed on J0 in 99.88s, 1 test, 1 failure <<< FAILURES!
[...truncated 485 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:190: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:443: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1245: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:889: There were test failures: 312 suites, 1316 tests, 1 failure, 20 ignored (13 assumptions)
Total time: 39 minutes 35 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_25 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0-ea-b94) - Build #
6351 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6351/
Java: 32bit/jdk1.8.0-ea-b94 -client -XX:+UseSerialGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ShardSplitTest
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.ShardSplitTest: 1) Thread[id=2903, name=IPC Parameter Sending Thread #2, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:360) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:939) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.ShardSplitTest:
1) Thread[id=2903, name=IPC Parameter Sending Thread #2, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:360)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:939)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
at __randomizedtesting.SeedInfo.seed([575283B251B3D6A]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ShardSplitTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=2903, name=IPC Parameter Sending Thread #2, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:360) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:939) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=2903, name=IPC Parameter Sending Thread #2, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:360)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:939)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
at __randomizedtesting.SeedInfo.seed([575283B251B3D6A]:0)
Build Log:
[...truncated 9493 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ShardSplitTest
[junit4:junit4] 2> 331433 T2755 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /wcv/n
[junit4:junit4] 2> 331445 T2755 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-ShardSplitTest-1372428758045
[junit4:junit4] 2> 331446 T2755 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 331446 T2756 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 331546 T2755 oasc.ZkTestServer.run start zk server on port:33996
[junit4:junit4] 2> 331547 T2755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 331806 T2762 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2819e6 name:ZooKeeperConnection Watcher:127.0.0.1:33996 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 331806 T2755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 331807 T2755 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 331811 T2755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 331812 T2764 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@efff93 name:ZooKeeperConnection Watcher:127.0.0.1:33996/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 331812 T2755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 331812 T2755 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 331814 T2755 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 331816 T2755 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 331817 T2755 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 331819 T2755 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 331819 T2755 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 331822 T2755 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 331823 T2755 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 331825 T2755 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 331825 T2755 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 331827 T2755 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 331827 T2755 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 331829 T2755 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 331829 T2755 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 331831 T2755 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 331832 T2755 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 331833 T2755 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 331834 T2755 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 331835 T2755 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 331836 T2755 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 331837 T2755 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 331838 T2755 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 331906 T2755 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 331913 T2755 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40525
[junit4:junit4] 2> 331914 T2755 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 331914 T2755 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 331915 T2755 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-controljetty-1372428758440
[junit4:junit4] 2> 331915 T2755 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-controljetty-1372428758440/solr.xml
[junit4:junit4] 2> 331916 T2755 oasc.CoreContainer.<init> New CoreContainer 1967357
[junit4:junit4] 2> 331917 T2755 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-controljetty-1372428758440/'
[junit4:junit4] 2> 331917 T2755 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-controljetty-1372428758440/'
[junit4:junit4] 2> 331980 T2755 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 331981 T2755 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 331981 T2755 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 331981 T2755 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 331982 T2755 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 331982 T2755 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 331983 T2755 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 331983 T2755 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 331983 T2755 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 331984 T2755 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 331987 T2755 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 331987 T2755 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33996/solr
[junit4:junit4] 2> 331988 T2755 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 331989 T2755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 331990 T2775 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59a66f name:ZooKeeperConnection Watcher:127.0.0.1:33996 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 331990 T2755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 331992 T2755 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 331995 T2755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 331996 T2777 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ccb583 name:ZooKeeperConnection Watcher:127.0.0.1:33996/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 331997 T2755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 331998 T2755 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 332000 T2755 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 332002 T2755 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 332003 T2755 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40525_wcv%2Fn
[junit4:junit4] 2> 332004 T2755 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40525_wcv%2Fn
[junit4:junit4] 2> 332006 T2755 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 332009 T2755 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 332011 T2755 oasc.Overseer.start Overseer (id=89943491087433731-127.0.0.1:40525_wcv%2Fn-n_0000000000) starting
[junit4:junit4] 2> 332013 T2755 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 332016 T2779 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 332017 T2755 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 332019 T2755 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 332020 T2755 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 332023 T2778 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 332026 T2780 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 332026 T2780 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 332028 T2780 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 333526 T2778 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 333527 T2778 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40525_wcv%2Fn",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40525/wcv/n"}
[junit4:junit4] 2> 333528 T2778 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 333528 T2778 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 333531 T2777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 334028 T2780 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 334029 T2780 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-controljetty-1372428758440/collection1
[junit4:junit4] 2> 334029 T2780 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 334029 T2780 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 334030 T2780 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 334030 T2780 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-controljetty-1372428758440/collection1/'
[junit4:junit4] 2> 334031 T2780 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ShardSplitTest-controljetty-1372428758440/collection1/lib/README' to classloader
[junit4:junit4] 2> 334031 T2780 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ShardSplitTest-controljetty-1372428758440/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 334066 T2780 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 334106 T2780 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 334108 T2780 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 334114 T2780 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 334493 T2780 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 334496 T2780 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 334498 T2780 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 334501 T2780 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 334517 T2780 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 334517 T2780 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-controljetty-1372428758440/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1372428758045/control/data/
[junit4:junit4] 2> 334517 T2780 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@119c41f
[junit4:junit4] 2> 334519 T2780 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1372428758045/control/data
[junit4:junit4] 2> 334519 T2780 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1372428758045/control/data/index/
[junit4:junit4] 2> 334520 T2780 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1372428758045/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 334520 T2780 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1372428758045/control/data/index
[junit4:junit4] 2> 334521 T2780 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5f92f4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4f47e2),segFN=segments_1,generation=1}
[junit4:junit4] 2> 334522 T2780 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 334523 T2780 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 334523 T2780 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 334524 T2780 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 334524 T2780 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 334525 T2780 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 334525 T2780 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 334525 T2780 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 334526 T2780 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 334526 T2780 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 334528 T2780 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 334530 T2780 oass.SolrIndexSearcher.<init> Opening Searcher@d4fddd main
[junit4:junit4] 2> 334531 T2780 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 334531 T2780 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 334535 T2781 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d4fddd main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 334536 T2780 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 334536 T2780 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40525/wcv/n collection:control_collection shard:shard1
[junit4:junit4] 2> 334536 T2780 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 334541 T2780 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4] 2> 334543 T2780 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 334543 T2780 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 334543 T2780 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40525/wcv/n/collection1/
[junit4:junit4] 2> 334544 T2780 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 334544 T2780 oasc.SyncStrategy.syncToMe http://127.0.0.1:40525/wcv/n/collection1/ has no replicas
[junit4:junit4] 2> 334544 T2780 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40525/wcv/n/collection1/ shard1
[junit4:junit4] 2> 334544 T2780 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 335033 T2778 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 335039 T2777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 335051 T2780 oasc.ZkController.register We are http://127.0.0.1:40525/wcv/n/collection1/ and leader is http://127.0.0.1:40525/wcv/n/collection1/
[junit4:junit4] 2> 335052 T2780 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40525/wcv/n
[junit4:junit4] 2> 335052 T2780 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 335052 T2780 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 335052 T2780 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 335055 T2780 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 335055 T2755 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 335056 T2755 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 335056 T2755 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 335059 T2755 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 335061 T2755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 335063 T2784 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10b964c name:ZooKeeperConnection Watcher:127.0.0.1:33996/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 335063 T2755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 335064 T2755 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 335065 T2755 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 335133 T2755 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 335135 T2755 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55944
[junit4:junit4] 2> 335136 T2755 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 335137 T2755 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 335137 T2755 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty1-1372428761666
[junit4:junit4] 2> 335137 T2755 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty1-1372428761666/solr.xml
[junit4:junit4] 2> 335138 T2755 oasc.CoreContainer.<init> New CoreContainer 7472035
[junit4:junit4] 2> 335138 T2755 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty1-1372428761666/'
[junit4:junit4] 2> 335138 T2755 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty1-1372428761666/'
[junit4:junit4] 2> 335190 T2755 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 335191 T2755 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 335191 T2755 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 335192 T2755 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 335192 T2755 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 335192 T2755 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 335193 T2755 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 335193 T2755 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 335193 T2755 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 335194 T2755 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 335197 T2755 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 335197 T2755 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33996/solr
[junit4:junit4] 2> 335198 T2755 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 335198 T2755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 335200 T2795 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b5d300 name:ZooKeeperConnection Watcher:127.0.0.1:33996 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 335201 T2755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 335202 T2755 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 335205 T2755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 335207 T2797 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@225c7a name:ZooKeeperConnection Watcher:127.0.0.1:33996/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 335207 T2755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 335210 T2755 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 336212 T2755 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55944_wcv%2Fn
[junit4:junit4] 2> 336214 T2755 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55944_wcv%2Fn
[junit4:junit4] 2> 336216 T2784 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 336216 T2797 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 336216 T2777 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 336217 T2777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 336221 T2798 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 336221 T2798 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 336222 T2798 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 336543 T2778 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 336545 T2778 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40525_wcv%2Fn",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40525/wcv/n"}
[junit4:junit4] 2> 336548 T2778 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55944_wcv%2Fn",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55944/wcv/n"}
[junit4:junit4] 2> 336548 T2778 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 336548 T2778 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 336551 T2784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 336551 T2777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 336551 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 337223 T2798 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 337223 T2798 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty1-1372428761666/collection1
[junit4:junit4] 2> 337223 T2798 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 337224 T2798 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 337224 T2798 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 337225 T2798 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty1-1372428761666/collection1/'
[junit4:junit4] 2> 337226 T2798 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ShardSplitTest-jetty1-1372428761666/collection1/lib/README' to classloader
[junit4:junit4] 2> 337226 T2798 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ShardSplitTest-jetty1-1372428761666/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 337255 T2798 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 337293 T2798 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 337294 T2798 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 337298 T2798 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 337748 T2798 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 337751 T2798 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 337753 T2798 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 337756 T2798 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 337773 T2798 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 337773 T2798 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty1-1372428761666/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty1/
[junit4:junit4] 2> 337773 T2798 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@119c41f
[junit4:junit4] 2> 337774 T2798 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty1
[junit4:junit4] 2> 337775 T2798 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty1/index/
[junit4:junit4] 2> 337775 T2798 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 337776 T2798 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty1/index
[junit4:junit4] 2> 337777 T2798 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5438f1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@12a059d),segFN=segments_1,generation=1}
[junit4:junit4] 2> 337777 T2798 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 337779 T2798 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 337779 T2798 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 337780 T2798 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 337780 T2798 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 337781 T2798 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 337781 T2798 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 337781 T2798 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 337782 T2798 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 337782 T2798 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 337784 T2798 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 337787 T2798 oass.SolrIndexSearcher.<init> Opening Searcher@f3d43 main
[junit4:junit4] 2> 337787 T2798 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 337787 T2798 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 337795 T2799 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f3d43 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 337798 T2798 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 337798 T2798 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55944/wcv/n collection:collection1 shard:shard1
[junit4:junit4] 2> 337799 T2798 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 337869 T2798 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4] 2> 337871 T2798 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 337871 T2798 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 337872 T2798 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55944/wcv/n/collection1/
[junit4:junit4] 2> 337872 T2798 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 337872 T2798 oasc.SyncStrategy.syncToMe http://127.0.0.1:55944/wcv/n/collection1/ has no replicas
[junit4:junit4] 2> 337872 T2798 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55944/wcv/n/collection1/ shard1
[junit4:junit4] 2> 337873 T2798 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 338056 T2778 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 338062 T2784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 338062 T2777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 338062 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 338080 T2798 oasc.ZkController.register We are http://127.0.0.1:55944/wcv/n/collection1/ and leader is http://127.0.0.1:55944/wcv/n/collection1/
[junit4:junit4] 2> 338080 T2798 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55944/wcv/n
[junit4:junit4] 2> 338080 T2798 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 338081 T2798 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 338081 T2798 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 338084 T2798 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 338086 T2755 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 338087 T2755 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 338088 T2755 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 338187 T2755 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 338189 T2755 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49506
[junit4:junit4] 2> 338190 T2755 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 338190 T2755 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 338190 T2755 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty2-1372428764698
[junit4:junit4] 2> 338191 T2755 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty2-1372428764698/solr.xml
[junit4:junit4] 2> 338191 T2755 oasc.CoreContainer.<init> New CoreContainer 31672696
[junit4:junit4] 2> 338192 T2755 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty2-1372428764698/'
[junit4:junit4] 2> 338192 T2755 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty2-1372428764698/'
[junit4:junit4] 2> 338247 T2755 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 338248 T2755 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 338248 T2755 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 338248 T2755 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 338249 T2755 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 338249 T2755 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 338249 T2755 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 338250 T2755 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 338250 T2755 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 338251 T2755 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 338254 T2755 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 338254 T2755 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33996/solr
[junit4:junit4] 2> 338255 T2755 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 338255 T2755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 338257 T2811 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d42ae2 name:ZooKeeperConnection Watcher:127.0.0.1:33996 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 338257 T2755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 338259 T2755 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 338263 T2755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 338264 T2813 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@199865e name:ZooKeeperConnection Watcher:127.0.0.1:33996/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 338265 T2755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 338268 T2755 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 339271 T2755 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49506_wcv%2Fn
[junit4:junit4] 2> 339272 T2755 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49506_wcv%2Fn
[junit4:junit4] 2> 339274 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 339274 T2777 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 339274 T2813 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 339274 T2784 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 339275 T2777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 339275 T2784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 339276 T2797 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 339389 T2814 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 339389 T2814 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 339391 T2814 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 339566 T2778 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 339567 T2778 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55944_wcv%2Fn",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55944/wcv/n"}
[junit4:junit4] 2> 339570 T2778 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49506_wcv%2Fn",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49506/wcv/n"}
[junit4:junit4] 2> 339571 T2778 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 339571 T2778 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4] 2> 339574 T2784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 339574 T2777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 339574 T2813 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 339574 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 340391 T2814 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 340392 T2814 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty2-1372428764698/collection1
[junit4:junit4] 2> 340392 T2814 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 340393 T2814 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 340393 T2814 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 340394 T2814 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty2-1372428764698/collection1/'
[junit4:junit4] 2> 340395 T2814 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ShardSplitTest-jetty2-1372428764698/collection1/lib/README' to classloader
[junit4:junit4] 2> 340396 T2814 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ShardSplitTest-jetty2-1372428764698/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 340444 T2814 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 340518 T2814 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 340519 T2814 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 340525 T2814 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 340909 T2814 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 340912 T2814 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 340913 T2814 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 340916 T2814 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 340932 T2814 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 340932 T2814 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty2-1372428764698/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty2/
[junit4:junit4] 2> 340933 T2814 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@119c41f
[junit4:junit4] 2> 340934 T2814 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty2
[junit4:junit4] 2> 340934 T2814 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty2/index/
[junit4:junit4] 2> 340934 T2814 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 340935 T2814 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty2/index
[junit4:junit4] 2> 340936 T2814 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1ca63 lockFactory=org.apache.lucene.store.NativeFSLockFactory@9c2f8c),segFN=segments_1,generation=1}
[junit4:junit4] 2> 340937 T2814 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 340938 T2814 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 340939 T2814 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 340939 T2814 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 340940 T2814 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 340941 T2814 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 340941 T2814 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 340941 T2814 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 340942 T2814 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 340942 T2814 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 340944 T2814 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 340947 T2814 oass.SolrIndexSearcher.<init> Opening Searcher@c76384 main
[junit4:junit4] 2> 340947 T2814 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 340947 T2814 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 340953 T2815 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c76384 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 340955 T2814 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 340955 T2814 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49506/wcv/n collection:collection1 shard:shard2
[junit4:junit4] 2> 340956 T2814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 340960 T2814 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4:junit4] 2> 340961 T2814 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 340961 T2814 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 340962 T2814 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49506/wcv/n/collection1/
[junit4:junit4] 2> 340962 T2814 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 340962 T2814 oasc.SyncStrategy.syncToMe http://127.0.0.1:49506/wcv/n/collection1/ has no replicas
[junit4:junit4] 2> 340962 T2814 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49506/wcv/n/collection1/ shard2
[junit4:junit4] 2> 340963 T2814 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 341079 T2778 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 341086 T2777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 341086 T2813 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 341086 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 341086 T2784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 341136 T2814 oasc.ZkController.register We are http://127.0.0.1:49506/wcv/n/collection1/ and leader is http://127.0.0.1:49506/wcv/n/collection1/
[junit4:junit4] 2> 341136 T2814 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49506/wcv/n
[junit4:junit4] 2> 341137 T2814 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 341137 T2814 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 341137 T2814 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 341138 T2814 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 341139 T2755 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 341140 T2755 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 341140 T2755 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 341255 T2755 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 341256 T2755 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40659
[junit4:junit4] 2> 341257 T2755 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 341258 T2755 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 341258 T2755 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty3-1372428767745
[junit4:junit4] 2> 341258 T2755 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty3-1372428767745/solr.xml
[junit4:junit4] 2> 341259 T2755 oasc.CoreContainer.<init> New CoreContainer 20420716
[junit4:junit4] 2> 341259 T2755 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty3-1372428767745/'
[junit4:junit4] 2> 341260 T2755 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty3-1372428767745/'
[junit4:junit4] 2> 341311 T2755 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 341311 T2755 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 341312 T2755 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 341312 T2755 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 341312 T2755 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 341313 T2755 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 341313 T2755 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 341313 T2755 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 341314 T2755 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 341314 T2755 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 341317 T2755 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 341318 T2755 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33996/solr
[junit4:junit4] 2> 341318 T2755 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 341319 T2755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 341320 T2827 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@123316c name:ZooKeeperConnection Watcher:127.0.0.1:33996 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 341321 T2755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 341323 T2755 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 341326 T2755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 341327 T2829 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d9e0d name:ZooKeeperConnection Watcher:127.0.0.1:33996/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 341327 T2755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 341330 T2755 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 342332 T2755 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40659_wcv%2Fn
[junit4:junit4] 2> 342333 T2755 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40659_wcv%2Fn
[junit4:junit4] 2> 342335 T2813 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 342335 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 342336 T2784 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 342336 T2784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 342336 T2777 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 342337 T2777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 342337 T2829 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 342338 T2813 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 342338 T2797 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 342343 T2830 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 342343 T2830 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 342344 T2830 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 342592 T2778 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 342593 T2778 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49506_wcv%2Fn",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49506/wcv/n"}
[junit4:junit4] 2> 342595 T2778 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node3",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40659_wcv%2Fn",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40659/wcv/n"}
[junit4:junit4] 2> 342596 T2778 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 342596 T2778 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 342639 T2829 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 342639 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 342639 T2777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 342639 T2813 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 342639 T2784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 343344 T2830 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 343345 T2830 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty3-1372428767745/collection1
[junit4:junit4] 2> 343345 T2830 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 343345 T2830 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 343346 T2830 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 343347 T2830 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty3-1372428767745/collection1/'
[junit4:junit4] 2> 343347 T2830 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ShardSplitTest-jetty3-1372428767745/collection1/lib/README' to classloader
[junit4:junit4] 2> 343348 T2830 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ShardSplitTest-jetty3-1372428767745/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 343390 T2830 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 343452 T2830 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 343454 T2830 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 343460 T2830 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 343873 T2830 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 343876 T2830 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 343878 T2830 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 343880 T2830 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 343897 T2830 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 343897 T2830 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty3-1372428767745/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty3/
[junit4:junit4] 2> 343897 T2830 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@119c41f
[junit4:junit4] 2> 343899 T2830 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty3
[junit4:junit4] 2> 343899 T2830 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty3/index/
[junit4:junit4] 2> 343899 T2830 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 343900 T2830 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty3/index
[junit4:junit4] 2> 343901 T2830 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4e22e9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@95a84c),segFN=segments_1,generation=1}
[junit4:junit4] 2> 343901 T2830 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 343903 T2830 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 343903 T2830 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 343904 T2830 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 343905 T2830 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 343905 T2830 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 343905 T2830 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 343906 T2830 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 343906 T2830 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 343907 T2830 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 343908 T2830 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 343911 T2830 oass.SolrIndexSearcher.<init> Opening Searcher@108dde0 main
[junit4:junit4] 2> 343912 T2830 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 343912 T2830 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 343917 T2831 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@108dde0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 343920 T2830 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 343920 T2830 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40659/wcv/n collection:collection1 shard:shard1
[junit4:junit4] 2> 343922 T2830 oasc.ZkController.register We are http://127.0.0.1:40659/wcv/n/collection1/ and leader is http://127.0.0.1:55944/wcv/n/collection1/
[junit4:junit4] 2> 343922 T2830 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40659/wcv/n
[junit4:junit4] 2> 343922 T2830 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 343923 T2830 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C775 name=collection1 org.apache.solr.core.SolrCore@6f4a31 url=http://127.0.0.1:40659/wcv/n/collection1 node=127.0.0.1:40659_wcv%2Fn C775_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:40659_wcv%2Fn, base_url=http://127.0.0.1:40659/wcv/n}
[junit4:junit4] 2> 343923 T2832 C775 P40659 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 343923 T2830 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 343923 T2832 C775 P40659 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 343924 T2832 C775 P40659 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 343924 T2832 C775 P40659 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 343938 T2755 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 343938 T2755 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> ASYNC NEW_CORE C776 name=collection1 org.apache.solr.core.SolrCore@6f4a31 url=http://127.0.0.1:40659/wcv/n/collection1 node=127.0.0.1:40659_wcv%2Fn C776_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:40659_wcv%2Fn, base_url=http://127.0.0.1:40659/wcv/n}
[junit4:junit4] 2> 343938 T2832 C776 P40659 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 343939 T2755 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 343943 T2790 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 344013 T2755 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 344015 T2755 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42486
[junit4:junit4] 2> 344016 T2755 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 344017 T2755 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 344017 T2755 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty4-1372428770542
[junit4:junit4] 2> 344017 T2755 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty4-1372428770542/solr.xml
[junit4:junit4] 2> 344018 T2755 oasc.CoreContainer.<init> New CoreContainer 17838386
[junit4:junit4] 2> 344018 T2755 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty4-1372428770542/'
[junit4:junit4] 2> 344018 T2755 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty4-1372428770542/'
[junit4:junit4] 2> 344069 T2755 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 344070 T2755 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 344070 T2755 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 344070 T2755 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 344071 T2755 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 344071 T2755 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 344071 T2755 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 344072 T2755 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 344072 T2755 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 344072 T2755 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 344075 T2755 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 344076 T2755 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33996/solr
[junit4:junit4] 2> 344076 T2755 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 344077 T2755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 344079 T2844 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8ccd77 name:ZooKeeperConnection Watcher:127.0.0.1:33996 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 344079 T2755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 344081 T2755 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 344084 T2755 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 344085 T2846 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b37f4c name:ZooKeeperConnection Watcher:127.0.0.1:33996/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 344085 T2755 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 344088 T2755 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 344144 T2778 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 344145 T2778 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node3",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40659_wcv%2Fn",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40659/wcv/n"}
[junit4:junit4] 2> 344155 T2784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 344156 T2777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 344156 T2846 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 344156 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 344155 T2813 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 344155 T2829 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 344944 T2790 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 344944 T2790 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node3&state=recovering&nodeName=127.0.0.1:40659_wcv%252Fn&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 345091 T2755 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42486_wcv%2Fn
[junit4:junit4] 2> 345092 T2755 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42486_wcv%2Fn
[junit4:junit4] 2> 345094 T2829 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 345094 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 345094 T2846 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 345094 T2813 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 345094 T2777 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 345094 T2784 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 345095 T2777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 345095 T2784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 345096 T2829 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 345096 T2797 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 345096 T2813 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 345097 T2846 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 345100 T2847 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 345101 T2847 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 345102 T2847 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 345661 T2778 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 345662 T2778 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42486_wcv%2Fn",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42486/wcv/n"}
[junit4:junit4] 2> 345662 T2778 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 345662 T2778 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4] 2> 345665 T2813 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 345666 T2784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 345666 T2846 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 345666 T2829 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 345665 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 345665 T2777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 346102 T2847 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 346103 T2847 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty4-1372428770542/collection1
[junit4:junit4] 2> 346103 T2847 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 346103 T2847 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 346104 T2847 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 346105 T2847 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty4-1372428770542/collection1/'
[junit4:junit4] 2> 346105 T2847 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ShardSplitTest-jetty4-1372428770542/collection1/lib/README' to classloader
[junit4:junit4] 2> 346106 T2847 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ShardSplitTest-jetty4-1372428770542/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 346137 T2847 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 346176 T2847 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 346177 T2847 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 346181 T2847 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 346596 T2847 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 346599 T2847 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 346600 T2847 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 346604 T2847 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 346621 T2847 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 346622 T2847 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ShardSplitTest-jetty4-1372428770542/collection1/, dataDir=./org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty4/
[junit4:junit4] 2> 346622 T2847 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@119c41f
[junit4:junit4] 2> 346623 T2847 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty4
[junit4:junit4] 2> 346623 T2847 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty4/index/
[junit4:junit4] 2> 346623 T2847 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 346624 T2847 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.ShardSplitTest-1372428758045/jetty4/index
[junit4:junit4] 2> 346625 T2847 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1267457 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e37219),segFN=segments_1,generation=1}
[junit4:junit4] 2> 346626 T2847 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 346627 T2847 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 346628 T2847 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 346628 T2847 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 346629 T2847 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 346629 T2847 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 346630 T2847 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 346630 T2847 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 346631 T2847 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 346631 T2847 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 346632 T2847 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 346635 T2847 oass.SolrIndexSearcher.<init> Opening Searcher@9ef68b main
[junit4:junit4] 2> 346636 T2847 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 346636 T2847 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 346642 T2848 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9ef68b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 346644 T2847 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 346644 T2847 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42486/wcv/n collection:collection1 shard:shard2
[junit4:junit4] 2> 346648 T2847 oasc.ZkController.register We are http://127.0.0.1:42486/wcv/n/collection1/ and leader is http://127.0.0.1:49506/wcv/n/collection1/
[junit4:junit4] 2> 346648 T2847 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42486/wcv/n
[junit4:junit4] 2> 346648 T2847 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 346648 T2847 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C777 name=collection1 org.apache.solr.core.SolrCore@14b668c url=http://127.0.0.1:42486/wcv/n/collection1 node=127.0.0.1:42486_wcv%2Fn C777_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:42486_wcv%2Fn, base_url=http://127.0.0.1:42486/wcv/n}
[junit4:junit4] 2> 346649 T2849 C777 P42486 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 346649 T2847 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 346649 T2849 C777 P42486 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 346650 T2849 C777 P42486 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 346650 T2849 C777 P42486 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 346650 T2755 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 346651 T2755 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 346651 T2755 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 346655 T2755 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 346656 T2849 C777 P42486 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 346657 T2755 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 346657 T2755 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 346658 T2755 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 346660 T2809 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> ASYNC NEW_CORE C778 name=collection1 org.apache.solr.core.SolrCore@6f4a31 url=http://127.0.0.1:40659/wcv/n/collection1 node=127.0.0.1:40659_wcv%2Fn C778_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:40659_wcv%2Fn, base_url=http://127.0.0.1:40659/wcv/n}
[junit4:junit4] 2> 346945 T2832 C778 P40659 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:55944/wcv/n/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 346946 T2832 C778 P40659 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:40659/wcv/n START replicas=[http://127.0.0.1:55944/wcv/n/collection1/] nUpdates=100
[junit4:junit4] 2> 346946 T2832 C778 P40659 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 346947 T2832 C778 P40659 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 346947 T2832 C778 P40659 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 346947 T2832 C778 P40659 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 346947 T2832 C778 P40659 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 346948 T2832 C778 P40659 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:55944/wcv/n/collection1/. core=collection1
[junit4:junit4] 2> 346948 T2832 C778 P40659 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C779 name=collection1 org.apache.solr.core.SolrCore@f3a7ee url=http://127.0.0.1:55944/wcv/n/collection1 node=127.0.0.1:55944_wcv%2Fn C779_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:55944_wcv%2Fn, base_url=http://127.0.0.1:55944/wcv/n, leader=true}
[junit4:junit4] 2> 346953 T2790 C779 P55944 oasc.SolrCore.execute [collection1] webapp=/wcv/n path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 346958 T2793 C779 P55944 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 346960 T2793 C779 P55944 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5438f1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@12a059d),segFN=segments_1,generation=1}
[junit4:junit4] 2> 346961 T2793 C779 P55944 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 346962 T2793 C779 P55944 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 346962 T2793 C779 P55944 oass.SolrIndexSearcher.<init> Opening Searcher@1c485f3 realtime
[junit4:junit4] 2> 346963 T2793 C779 P55944 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 346963 T2793 C779 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 346964 T2832 C778 P40659 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 346965 T2832 C778 P40659 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 346970 T2792 C779 P55944 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 346970 T2792 C779 P55944 oasc.SolrCore.execute [collection1] webapp=/wcv/n path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=4
[junit4:junit4] 2> 346971 T2832 C778 P40659 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 346971 T2832 C778 P40659 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 346972 T2832 C778 P40659 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 346972 T2832 C778 P40659 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 346974 T2832 C778 P40659 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 347170 T2778 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 347171 T2778 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42486_wcv%2Fn",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42486/wcv/n"}
[junit4:junit4] 2> 347173 T2778 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node3",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40659_wcv%2Fn",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40659/wcv/n"}
[junit4:junit4] 2> 347176 T2846 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 347176 T2829 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 347176 T2777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 347176 T2784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 347176 T2813 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 347176 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 347660 T2755 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 347661 T2809 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 347661 T2809 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node4&state=recovering&nodeName=127.0.0.1:42486_wcv%252Fn&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 348661 T2755 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C777_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:42486_wcv%2Fn, base_url=http://127.0.0.1:42486/wcv/n}
[junit4:junit4] 2> 349662 T2849 C777 P42486 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:49506/wcv/n/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 349663 T2849 C777 P42486 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:42486/wcv/n START replicas=[http://127.0.0.1:49506/wcv/n/collection1/] nUpdates=100
[junit4:junit4] 2> 349663 T2755 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 349663 T2849 C777 P42486 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 349664 T2849 C777 P42486 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 349664 T2849 C777 P42486 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 349664 T2849 C777 P42486 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 349664 T2849 C777 P42486 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 349664 T2849 C777 P42486 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:49506/wcv/n/collection1/. core=collection1
[junit4:junit4] 2> 349665 T2849 C777 P42486 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C780 name=collection1 org.apache.solr.core.SolrCore@8fd0dc url=http://127.0.0.1:49506/wcv/n/collection1 node=127.0.0.1:49506_wcv%2Fn C780_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:49506_wcv%2Fn, base_url=http://127.0.0.1:49506/wcv/n, leader=true}
[junit4:junit4] 2> 349672 T2808 C780 P49506 oasc.SolrCore.execute [collection1] webapp=/wcv/n path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 349675 T2807 C780 P49506 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 349677 T2807 C780 P49506 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1ca63 lockFactory=org.apache.lucene.store.NativeFSLockFactory@9c2f8c),segFN=segments_1,generation=1}
[junit4:junit4] 2> 349678 T2807 C780 P49506 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 349679 T2807 C780 P49506 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 349679 T2807 C780 P49506 oass.SolrIndexSearcher.<init> Opening Searcher@13baa0d realtime
[junit4:junit4] 2> 349680 T2807 C780 P49506 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 349680 T2807 C780 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 349681 T2849 C777 P42486 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 349681 T2849 C777 P42486 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 349683 T2808 C780 P49506 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 349684 T2808 C780 P49506 oasc.SolrCore.execute [collection1] webapp=/wcv/n path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 349685 T2849 C777 P42486 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 349685 T2849 C777 P42486 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 349685 T2849 C777 P42486 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 349685 T2849 C777 P42486 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 349686 T2849 C777 P42486 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 350184 T2778 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 350184 T2778 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42486_wcv%2Fn",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42486/wcv/n"}
[junit4:junit4] 2> 350208 T2777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 350209 T2784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 350209 T2829 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 350209 T2797 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 350209 T2813 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 350208 T2846 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 350665 T2755 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 350666 T2755 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C781 name=collection1 org.apache.solr.core.SolrCore@f83ec1 url=http://127.0.0.1:40525/wcv/n/collection1 node=127.0.0.1:40525_wcv%2Fn C781_STATE=coll:control_collection core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:40525_wcv%2Fn, base_url=http://127.0.0.1:40525/wcv/n, leader=true}
[junit4:junit4] 2> 350672 T2771 C781 P40525 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 350674 T2771 C781 P40525 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5f92f4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4f47e2),segFN=segments_1,generation=1}
[junit4:junit4] 2> 350674 T2771 C781 P40525 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 350675 T2771 C781 P40525 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 350675 T2771 C781 P40525 oass.SolrIndexSearcher.<init> Opening Searcher@17a5633 main
[junit4:junit4] 2> 350676 T2771 C781 P40525 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 350691 T2781 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@17a5633 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> ASYNC NEW_CORE C782 name=collection1 org.apache.solr.core.SolrCore@f83ec1 url=http://127.0.0.1:40525/wcv/n/collection1 node=127.0.0.1:40525_wcv%2Fn C782_STATE=coll:control_collection core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:40525_wcv%2Fn, base_url=http://127.0.0.1:40525/wcv/n, leader=true}
[junit4:junit4] 2> 350692 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 21
[junit4:junit4] 2> ASYNC NEW_CORE C783 name=collection1 org.apache.solr.core.SolrCore@8fd0dc url=http://127.0.0.1:49506/wcv/n/collection1 node=127.0.0.1:49506_wcv%2Fn C783_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:49506_wcv%2Fn, base_url=http://127.0.0.1:49506/wcv/n, leader=true}
[junit4:junit4] 2> 350696 T2808 C783 P49506 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:55944/wcv/n/collection1/, StdNode: http://127.0.0.1:40659/wcv/n/collection1/, StdNode: http://127.0.0.1:49506/wcv/n/collection1/, StdNode: http://127.0.0.1:42486/wcv/n/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> ASYNC NEW_CORE C784 name=collection1 org.apache.solr.core.SolrCore@f3a7ee url=http://127.0.0.1:55944/wcv/n/collection1 node=127.0.0.1:55944_wcv%2Fn C784_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:55944_wcv%2Fn, base_url=http://127.0.0.1:55944/wcv/n, leader=true}
[junit4:junit4] 2> 350698 T2792 C784 P55944 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 350698 T2792 C784 P55944 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 350699 T2792 C784 P55944 oass.SolrIndexSearcher.<init> Opening Searcher@1936ec4 main
[junit4:junit4] 2> 350699 T2792 C784 P55944 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 350700 T2799 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1936ec4 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 350701 T2792 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4] 2> 350703 T2806 C783 P49506 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 350704 T2806 C783 P49506 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 350704 T2806 C783 P49506 oass.SolrIndexSearcher.<init> Opening Searcher@15fdf46 main
[junit4:junit4] 2> 350704 T2806 C783 P49506 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 350706 T2815 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@15fdf46 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 350706 T2806 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4] 2> ASYNC NEW_CORE C785 name=collection1 org.apache.solr.core.SolrCore@14b668c url=http://127.0.0.1:42486/wcv/n/collection1 node=127.0.0.1:42486_wcv%2Fn C785_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:42486_wcv%2Fn, base_url=http://127.0.0.1:42486/wcv/n}
[junit4:junit4] 2> 350708 T2839 C785 P42486 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 350710 T2839 C785 P42486 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1267457 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e37219),segFN=segments_1,generation=1}
[junit4:junit4] 2> 350711 T2839 C785 P42486 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> ASYNC NEW_CORE C786 name=collection1 org.apache.solr.core.SolrCore@6f4a31 url=http://127.0.0.1:40659/wcv/n/collection1 node=127.0.0.1:40659_wcv%2Fn C786_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:40659_wcv%2Fn, base_url=http://127.0.0.1:40659/wcv/n}
[junit4:junit4] 2> 350711 T2822 C786 P40659 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 350711 T2839 C785 P42486 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 350712 T2839 C785 P42486 oass.SolrIndexSearcher.<init> Opening Searcher@10535f5 main
[junit4:junit4] 2> 350713 T2822 C786 P40659 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4e22e9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@95a84c),segFN=segments_1,generation=1}
[junit4:junit4] 2> 350713 T2839 C785 P42486 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 350713 T2822 C786 P40659 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 350714 T2822 C786 P40659 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 350715 T2848 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@10535f5 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 350715 T2822 C786 P40659 oass.SolrIndexSearcher.<init> Opening Searcher@da2a0c main
[junit4:junit4] 2> 350716 T2822 C786 P40659 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 350716 T2839 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 350717 T2831 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@da2a0c main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 350717 T2822 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 350718 T2808 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 22
[junit4:junit4] 2> 350719 T2755 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 350726 T2793 C784 P55944 oasc.SolrCore.execute [collection1] webapp=/wcv/n path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 350729 T2824 C786 P40659 oasc.SolrCore.execute [collection1] webapp=/wcv/n path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 350731 T2807 C783 P49506 oasc.SolrCore.execute [collection1] webapp=/wcv/n path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 350733 T2842 C785 P42486 oasc.SolrCore.execute [collection1] webapp=/wcv/n path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 352739 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1439095879723122688)} 0 2
[junit4:junit4] 2> 352746 T2823 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={update.distrib=FROMLEADER&_version_=-1439095879726268416&update.from=http://127.0.0.1:55944/wcv/n/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439095879726268416)} 0 2
[junit4:junit4] 2> 352748 T2840 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={update.distrib=FROMLEADER&_version_=-1439095879729414144&update.from=http://127.0.0.1:49506/wcv/n/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439095879729414144)} 0 1
[junit4:junit4] 2> 352749 T2806 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1439095879729414144)} 0 5
[junit4:junit4] 2> 352750 T2793 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1439095879726268416)} 0 9
[junit4:junit4] 2> 352753 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[a!0 (1439095879737802752)]} 0 1
[junit4:junit4] 2> 352758 T2841 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[a!0 (1439095879740948480)]} 0 0
[junit4:junit4] 2> 352759 T2808 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[a!0 (1439095879740948480)]} 0 4
[junit4:junit4] 2> 352761 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[b!1 (1439095879747239936)]} 0 0
[junit4:junit4] 2> 352769 T2822 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[b!1 (1439095879752482816)]} 0 1
[junit4:junit4] 2> 352770 T2791 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[b!1 (1439095879752482816)]} 0 4
[junit4:junit4] 2> 352770 T2809 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[b!1]} 0 6
[junit4:junit4] 2> 352773 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[c!2 (1439095879759822848)]} 0 0
[junit4:junit4] 2> 352778 T2825 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[c!2 (1439095879762968576)]} 0 0
[junit4:junit4] 2> 352779 T2792 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[c!2 (1439095879762968576)]} 0 3
[junit4:junit4] 2> 352781 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[d!3 (1439095879768211456)]} 0 0
[junit4:junit4] 2> 352786 T2839 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[d!3 (1439095879771357184)]} 0 0
[junit4:junit4] 2> 352787 T2807 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[d!3 (1439095879771357184)]} 0 3
[junit4:junit4] 2> 352789 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[e!4 (1439095879776600064)]} 0 0
[junit4:junit4] 2> 352796 T2842 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[e!4 (1439095879781842944)]} 0 1
[junit4:junit4] 2> 352797 T2806 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[e!4 (1439095879781842944)]} 0 4
[junit4:junit4] 2> 352797 T2790 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[e!4]} 0 5
[junit4:junit4] 2> 352800 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[f!5 (1439095879787085824)]} 0 1
[junit4:junit4] 2> 352804 T2840 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[f!5 (1439095879790231552)]} 0 0
[junit4:junit4] 2> 352805 T2808 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[f!5 (1439095879790231552)]} 0 3
[junit4:junit4] 2> 352808 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[g!6 (1439095879796523008)]} 0 0
[junit4:junit4] 2> 352815 T2824 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[g!6 (1439095879800717312)]} 0 0
[junit4:junit4] 2> 352816 T2793 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[g!6 (1439095879800717312)]} 0 4
[junit4:junit4] 2> 352816 T2809 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[g!6]} 0 6
[junit4:junit4] 2> 352819 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[h!7 (1439095879808057344)]} 0 0
[junit4:junit4] 2> 352824 T2823 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[h!7 (1439095879811203072)]} 0 0
[junit4:junit4] 2> 352825 T2791 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[h!7 (1439095879811203072)]} 0 3
[junit4:junit4] 2> 352828 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[i!8 (1439095879817494528)]} 0 0
[junit4:junit4] 2> 352838 T2822 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[i!8 (1439095879824834560)]} 0 1
[junit4:junit4] 2> 352838 T2792 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[i!8 (1439095879824834560)]} 0 3
[junit4:junit4] 2> 352839 T2807 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[i!8]} 0 8
[junit4:junit4] 2> 352842 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[j!9 (1439095879832174592)]} 0 1
[junit4:junit4] 2> 352849 T2825 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[j!9 (1439095879836368896)]} 0 1
[junit4:junit4] 2> 352849 T2790 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[j!9 (1439095879836368896)]} 0 3
[junit4:junit4] 2> 352850 T2806 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[j!9]} 0 6
[junit4:junit4] 2> 352853 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[k!10 (1439095879843708928)]} 0 1
[junit4:junit4] 2> 352860 T2824 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[k!10 (1439095879848951808)]} 0 0
[junit4:junit4] 2> 352861 T2793 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[k!10 (1439095879848951808)]} 0 3
[junit4:junit4] 2> 352862 T2808 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[k!10]} 0 6
[junit4:junit4] 2> 352864 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[l!11 (1439095879855243264)]} 0 0
[junit4:junit4] 2> 352870 T2841 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[l!11 (1439095879858388992)]} 0 1
[junit4:junit4] 2> 352870 T2809 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[l!11 (1439095879858388992)]} 0 3
[junit4:junit4] 2> 352873 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[m!12 (1439095879864680448)]} 0 1
[junit4:junit4] 2> 352880 T2839 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[m!12 (1439095879868874752)]} 0 1
[junit4:junit4] 2> 352881 T2807 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[m!12 (1439095879868874752)]} 0 4
[junit4:junit4] 2> 352881 T2791 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[m!12]} 0 6
[junit4:junit4] 2> 352885 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[n!13 (1439095879876214784)]} 0 1
[junit4:junit4] 2> 352890 T2823 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[n!13 (1439095879880409088)]} 0 0
[junit4:junit4] 2> 352891 T2792 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[n!13 (1439095879880409088)]} 0 4
[junit4:junit4] 2> 352894 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[o!14 (1439095879886700544)]} 0 1
[junit4:junit4] 2> 352899 T2842 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[o!14 (1439095879889846272)]} 0 0
[junit4:junit4] 2> 352900 T2806 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[o!14 (1439095879889846272)]} 0 4
[junit4:junit4] 2> 352902 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[p!15 (1439095879895089152)]} 0 0
[junit4:junit4] 2> 352907 T2822 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[p!15 (1439095879898234880)]} 0 0
[junit4:junit4] 2> 352908 T2790 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[p!15 (1439095879898234880)]} 0 3
[junit4:junit4] 2> 352911 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[q!16 (1439095879904526336)]} 0 0
[junit4:junit4] 2> 352918 T2825 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[q!16 (1439095879908720640)]} 0 1
[junit4:junit4] 2> 352919 T2793 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[q!16 (1439095879908720640)]} 0 4
[junit4:junit4] 2> 352919 T2808 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[q!16]} 0 6
[junit4:junit4] 2> 352922 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[r!17 (1439095879916060672)]} 0 0
[junit4:junit4] 2> 352927 T2840 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[r!17 (1439095879919206400)]} 0 0
[junit4:junit4] 2> 352929 T2809 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[r!17 (1439095879919206400)]} 0 4
[junit4:junit4] 2> 352931 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[s!18 (1439095879925497856)]} 0 0
[junit4:junit4] 2> 352938 T2824 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[s!18 (1439095879930740736)]} 0 0
[junit4:junit4] 2> 352939 T2791 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[s!18 (1439095879930740736)]} 0 4
[junit4:junit4] 2> 352940 T2807 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[s!18]} 0 7
[junit4:junit4] 2> 352943 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[t!19 (1439095879937032192)]} 0 1
[junit4:junit4] 2> 352948 T2823 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[t!19 (1439095879940177920)]} 0 1
[junit4:junit4] 2> 352948 T2792 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[t!19 (1439095879940177920)]} 0 3
[junit4:junit4] 2> 352951 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[u!20 (1439095879946469376)]} 0 0
[junit4:junit4] 2> 352956 T2841 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[u!20 (1439095879948566528)]} 0 1
[junit4:junit4] 2> 352957 T2806 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[u!20 (1439095879948566528)]} 0 4
[junit4:junit4] 2> 352959 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[v!21 (1439095879954857984)]} 0 0
[junit4:junit4] 2> 352965 T2822 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[v!21 (1439095879959052288)]} 0 0
[junit4:junit4] 2> 352966 T2790 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[v!21 (1439095879959052288)]} 0 3
[junit4:junit4] 2> 352967 T2808 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[v!21]} 0 6
[junit4:junit4] 2> 352970 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[w!22 (1439095879965343744)]} 0 1
[junit4:junit4] 2> 352976 T2825 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[w!22 (1439095879970586624)]} 0 0
[junit4:junit4] 2> 352977 T2793 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[w!22 (1439095879970586624)]} 0 3
[junit4:junit4] 2> 352977 T2809 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[w!22]} 0 5
[junit4:junit4] 2> 352980 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[x!23 (1439095879976878080)]} 0 1
[junit4:junit4] 2> 352985 T2839 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[x!23 (1439095879978975232)]} 0 1
[junit4:junit4] 2> 352985 T2807 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[x!23 (1439095879978975232)]} 0 3
[junit4:junit4] 2> 352988 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[y!24 (1439095879985266688)]} 0 1
[junit4:junit4] 2> 352992 T2842 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[y!24 (1439095879987363840)]} 0 0
[junit4:junit4] 2> 352993 T2806 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[y!24 (1439095879987363840)]} 0 3
[junit4:junit4] 2> 352996 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[z!25 (1439095879992606720)]} 0 1
[junit4:junit4] 2> 353000 T2824 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[z!25 (1439095879995752448)]} 0 0
[junit4:junit4] 2> 353001 T2791 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[z!25 (1439095879995752448)]} 0 3
[junit4:junit4] 2> 353004 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[a!26 (1439095880000995328)]} 0 1
[junit4:junit4] 2> 353010 T2840 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[a!26 (1439095880006238208)]} 0 0
[junit4:junit4] 2> 353011 T2808 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[a!26 (1439095880006238208)]} 0 3
[junit4:junit4] 2> 353012 T2792 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[a!26]} 0 6
[junit4:junit4] 2> 353015 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[b!27 (1439095880012529664)]} 0 1
[junit4:junit4] 2> 353022 T2823 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[b!27 (1439095880017772544)]} 0 1
[junit4:junit4] 2> 353022 T2790 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[b!27 (1439095880017772544)]} 0 3
[junit4:junit4] 2> 353023 T2809 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[b!27]} 0 6
[junit4:junit4] 2> 353026 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[c!28 (1439095880024064000)]} 0 1
[junit4:junit4] 2> 353030 T2822 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[c!28 (1439095880027209728)]} 0 0
[junit4:junit4] 2> 353031 T2793 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[c!28 (1439095880027209728)]} 0 3
[junit4:junit4] 2> 353033 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[d!29 (1439095880032452608)]} 0 0
[junit4:junit4] 2> 353040 T2841 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[d!29 (1439095880037695488)]} 0 0
[junit4:junit4] 2> 353041 T2807 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[d!29 (1439095880037695488)]} 0 3
[junit4:junit4] 2> 353041 T2791 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[d!29]} 0 5
[junit4:junit4] 2> 353044 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[e!30 (1439095880043986944)]} 0 0
[junit4:junit4] 2> 353051 T2839 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[e!30 (1439095880049229824)]} 0 0
[junit4:junit4] 2> 353052 T2806 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[e!30 (1439095880049229824)]} 0 3
[junit4:junit4] 2> 353052 T2792 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[e!30]} 0 5
[junit4:junit4] 2> 353055 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[f!31 (1439095880054472704)]} 0 1
[junit4:junit4] 2> 353061 T2842 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[f!31 (1439095880059715584)]} 0 0
[junit4:junit4] 2> 353062 T2808 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[f!31 (1439095880059715584)]} 0 3
[junit4:junit4] 2> 353063 T2790 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[f!31]} 0 6
[junit4:junit4] 2> 353065 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[g!32 (1439095880066007040)]} 0 0
[junit4:junit4] 2> 353070 T2825 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[g!32 (1439095880069152768)]} 0 0
[junit4:junit4] 2> 353071 T2793 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[g!32 (1439095880069152768)]} 0 4
[junit4:junit4] 2> 353074 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[h!33 (1439095880075444224)]} 0 0
[junit4:junit4] 2> 353081 T2824 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[h!33 (1439095880080687104)]} 0 0
[junit4:junit4] 2> 353082 T2791 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[h!33 (1439095880080687104)]} 0 3
[junit4:junit4] 2> 353083 T2809 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[h!33]} 0 6
[junit4:junit4] 2> 353086 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[i!34 (1439095880086978560)]} 0 1
[junit4:junit4] 2> 353093 T2823 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[i!34 (1439095880092221440)]} 0 1
[junit4:junit4] 2> 353093 T2792 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[i!34 (1439095880092221440)]} 0 3
[junit4:junit4] 2> 353094 T2807 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[i!34]} 0 6
[junit4:junit4] 2> 353097 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[j!35 (1439095880099561472)]} 0 1
[junit4:junit4] 2> 353104 T2822 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[j!35 (1439095880104804352)]} 0 1
[junit4:junit4] 2> 353105 T2790 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[j!35 (1439095880104804352)]} 0 4
[junit4:junit4] 2> 353105 T2806 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[j!35]} 0 5
[junit4:junit4] 2> 353108 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[k!36 (1439095880111095808)]} 0 1
[junit4:junit4] 2> 353115 T2825 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[k!36 (1439095880115290112)]} 0 1
[junit4:junit4] 2> 353115 T2791 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[k!36 (1439095880115290112)]} 0 3
[junit4:junit4] 2> 353116 T2808 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[k!36]} 0 6
[junit4:junit4] 2> 353119 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[l!37 (1439095880122630144)]} 0 0
[junit4:junit4] 2> 353124 T2840 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[l!37 (1439095880125775872)]} 0 0
[junit4:junit4] 2> 353125 T2809 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[l!37 (1439095880125775872)]} 0 3
[junit4:junit4] 2> 353127 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[m!38 (1439095880131018752)]} 0 0
[junit4:junit4] 2> 353132 T2841 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[m!38 (1439095880134164480)]} 0 1
[junit4:junit4] 2> 353132 T2807 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[m!38 (1439095880134164480)]} 0 2
[junit4:junit4] 2> 353135 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[n!39 (1439095880139407360)]} 0 0
[junit4:junit4] 2> 353140 T2824 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[n!39 (1439095880142553088)]} 0 0
[junit4:junit4] 2> 353141 T2790 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[n!39 (1439095880142553088)]} 0 3
[junit4:junit4] 2> 353144 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[o!40 (1439095880148844544)]} 0 1
[junit4:junit4] 2> 353149 T2842 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[o!40 (1439095880150941696)]} 0 1
[junit4:junit4] 2> 353149 T2806 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[o!40 (1439095880150941696)]} 0 3
[junit4:junit4] 2> 353152 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[p!41 (1439095880157233152)]} 0 0
[junit4:junit4] 2> 353159 T2822 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[p!41 (1439095880162476032)]} 0 0
[junit4:junit4] 2> 353160 T2793 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[p!41 (1439095880162476032)]} 0 3
[junit4:junit4] 2> 353160 T2809 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[p!41]} 0 5
[junit4:junit4] 2> 353163 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[q!42 (1439095880168767488)]} 0 0
[junit4:junit4] 2> 353170 T2825 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[q!42 (1439095880172961792)]} 0 0
[junit4:junit4] 2> 353171 T2791 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[q!42 (1439095880172961792)]} 0 4
[junit4:junit4] 2> 353171 T2807 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[q!42]} 0 6
[junit4:junit4] 2> 353174 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[r!43 (1439095880180301824)]} 0 0
[junit4:junit4] 2> 353181 T2841 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[r!43 (1439095880185544704)]} 0 1
[junit4:junit4] 2> 353182 T2808 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[r!43 (1439095880185544704)]} 0 4
[junit4:junit4] 2> 353182 T2792 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[r!43]} 0 6
[junit4:junit4] 2> 353185 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[s!44 (1439095880191836160)]} 0 0
[junit4:junit4] 2> 353191 T2823 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[s!44 (1439095880196030464)]} 0 0
[junit4:junit4] 2> 353192 T2790 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[s!44 (1439095880196030464)]} 0 3
[junit4:junit4] 2> 353193 T2809 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[s!44]} 0 6
[junit4:junit4] 2> 353195 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[t!45 (1439095880202321920)]} 0 0
[junit4:junit4] 2> 353200 T2824 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[t!45 (1439095880205467648)]} 0 0
[junit4:junit4] 2> 353201 T2793 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[t!45 (1439095880205467648)]} 0 3
[junit4:junit4] 2> 353203 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[u!46 (1439095880210710528)]} 0 0
[junit4:junit4] 2> 353208 T2839 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[u!46 (1439095880213856256)]} 0 0
[junit4:junit4] 2> 353209 T2807 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[u!46 (1439095880213856256)]} 0 3
[junit4:junit4] 2> 353211 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[v!47 (1439095880219099136)]} 0 0
[junit4:junit4] 2> 353217 T2822 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[v!47 (1439095880222244864)]} 0 1
[junit4:junit4] 2> 353218 T2791 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[v!47 (1439095880222244864)]} 0 4
[junit4:junit4] 2> 353220 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[w!48 (1439095880228536320)]} 0 0
[junit4:junit4] 2> 353227 T2825 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[w!48 (1439095880233779200)]} 0 0
[junit4:junit4] 2> 353228 T2792 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[w!48 (1439095880233779200)]} 0 3
[junit4:junit4] 2> 353229 T2806 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[w!48]} 0 6
[junit4:junit4] 2> 353231 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[x!49 (1439095880240070656)]} 0 0
[junit4:junit4] 2> 353238 T2840 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[x!49 (1439095880245313536)]} 0 0
[junit4:junit4] 2> 353239 T2808 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[x!49 (1439095880245313536)]} 0 3
[junit4:junit4] 2> 353239 T2790 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[x!49]} 0 5
[junit4:junit4] 2> 353242 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[y!50 (1439095880250556416)]} 0 1
[junit4:junit4] 2> 353248 T2842 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[y!50 (1439095880255799296)]} 0 0
[junit4:junit4] 2> 353249 T2809 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[y!50 (1439095880255799296)]} 0 3
[junit4:junit4] 2> 353250 T2793 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[y!50]} 0 6
[junit4:junit4] 2> 353253 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[z!51 (1439095880262090752)]} 0 1
[junit4:junit4] 2> 353258 T2823 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[z!51 (1439095880265236480)]} 0 1
[junit4:junit4] 2> 353258 T2791 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[z!51 (1439095880265236480)]} 0 3
[junit4:junit4] 2> 353261 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[a!52 (1439095880271527936)]} 0 0
[junit4:junit4] 2> 353267 T2841 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[a!52 (1439095880274673664)]} 0 1
[junit4:junit4] 2> 353268 T2807 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[a!52 (1439095880274673664)]} 0 4
[junit4:junit4] 2> 353271 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[b!53 (1439095880280965120)]} 0 1
[junit4:junit4] 2> 353275 T2824 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[b!53 (1439095880284110848)]} 0 0
[junit4:junit4] 2> 353276 T2792 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[b!53 (1439095880284110848)]} 0 3
[junit4:junit4] 2> 353278 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[c!54 (1439095880289353728)]} 0 0
[junit4:junit4] 2> 353283 T2822 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[c!54 (1439095880292499456)]} 0 0
[junit4:junit4] 2> 353284 T2790 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[c!54 (1439095880292499456)]} 0 3
[junit4:junit4] 2> 353286 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[d!55 (1439095880297742336)]} 0 0
[junit4:junit4] 2> 353291 T2839 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[d!55 (1439095880300888064)]} 0 0
[junit4:junit4] 2> 353292 T2806 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[d!55 (1439095880300888064)]} 0 3
[junit4:junit4] 2> 353295 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[e!56 (1439095880306130944)]} 0 1
[junit4:junit4] 2> 353301 T2840 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[e!56 (1439095880311373824)]} 0 0
[junit4:junit4] 2> 353302 T2808 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[e!56 (1439095880311373824)]} 0 3
[junit4:junit4] 2> 353303 T2793 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[e!56]} 0 6
[junit4:junit4] 2> 353306 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[f!57 (1439095880317665280)]} 0 1
[junit4:junit4] 2> 353310 T2842 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[f!57 (1439095880320811008)]} 0 0
[junit4:junit4] 2> 353311 T2809 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[f!57 (1439095880320811008)]} 0 3
[junit4:junit4] 2> 353314 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[g!58 (1439095880326053888)]} 0 1
[junit4:junit4] 2> 353320 T2825 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[g!58 (1439095880331296768)]} 0 0
[junit4:junit4] 2> 353320 T2791 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[g!58 (1439095880331296768)]} 0 2
[junit4:junit4] 2> 353321 T2807 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[g!58]} 0 5
[junit4:junit4] 2> 353323 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[h!59 (1439095880336539648)]} 0 0
[junit4:junit4] 2> 353328 T2823 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[h!59 (1439095880339685376)]} 0 0
[junit4:junit4] 2> 353329 T2792 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[h!59 (1439095880339685376)]} 0 3
[junit4:junit4] 2> 353331 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[i!60 (1439095880344928256)]} 0 0
[junit4:junit4] 2> 353336 T2824 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[i!60 (1439095880348073984)]} 0 0
[junit4:junit4] 2> 353337 T2790 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[i!60 (1439095880348073984)]} 0 3
[junit4:junit4] 2> 353339 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[j!61 (1439095880353316864)]} 0 0
[junit4:junit4] 2> 353343 T2822 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[j!61 (1439095880355414016)]} 0 0
[junit4:junit4] 2> 353344 T2793 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[j!61 (1439095880355414016)]} 0 3
[junit4:junit4] 2> 353347 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[k!62 (1439095880361705472)]} 0 1
[junit4:junit4] 2> 353352 T2825 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[k!62 (1439095880363802624)]} 0 1
[junit4:junit4] 2> 353352 T2791 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[k!62 (1439095880363802624)]} 0 3
[junit4:junit4] 2> 353355 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[l!63 (1439095880369045504)]} 0 1
[junit4:junit4] 2> 353359 T2841 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[l!63 (1439095880372191232)]} 0 0
[junit4:junit4] 2> 353360 T2806 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[l!63 (1439095880372191232)]} 0 3
[junit4:junit4] 2> 353363 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[m!64 (1439095880378482688)]} 0 1
[junit4:junit4] 2> 353369 T2839 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[m!64 (1439095880382676992)]} 0 0
[junit4:junit4] 2> 353370 T2808 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[m!64 (1439095880382676992)]} 0 3
[junit4:junit4] 2> 353371 T2792 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[m!64]} 0 6
[junit4:junit4] 2> 353374 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[n!65 (1439095880388968448)]} 0 1
[junit4:junit4] 2> 353380 T2823 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[n!65 (1439095880394211328)]} 0 0
[junit4:junit4] 2> 353381 T2790 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[n!65 (1439095880394211328)]} 0 3
[junit4:junit4] 2> 353382 T2809 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[n!65]} 0 6
[junit4:junit4] 2> 353384 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[o!66 (1439095880400502784)]} 0 0
[junit4:junit4] 2> 353391 T2840 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[o!66 (1439095880405745664)]} 0 0
[junit4:junit4] 2> 353392 T2807 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[o!66 (1439095880405745664)]} 0 3
[junit4:junit4] 2> 353392 T2793 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[o!66]} 0 5
[junit4:junit4] 2> 353395 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[p!67 (1439095880412037120)]} 0 0
[junit4:junit4] 2> 353402 T2824 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[p!67 (1439095880417280000)]} 0 1
[junit4:junit4] 2> 353403 T2791 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[p!67 (1439095880417280000)]} 0 4
[junit4:junit4] 2> 353403 T2806 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[p!67]} 0 5
[junit4:junit4] 2> 353406 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[q!68 (1439095880423571456)]} 0 1
[junit4:junit4] 2> 353413 T2822 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[q!68 (1439095880427765760)]} 0 1
[junit4:junit4] 2> 353413 T2792 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[q!68 (1439095880427765760)]} 0 3
[junit4:junit4] 2> 353414 T2808 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[q!68]} 0 6
[junit4:junit4] 2> 353417 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[r!69 (1439095880434057216)]} 0 1
[junit4:junit4] 2> 353422 T2842 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[r!69 (1439095880437202944)]} 0 1
[junit4:junit4] 2> 353423 T2809 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[r!69 (1439095880437202944)]} 0 4
[junit4:junit4] 2> 353425 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[s!70 (1439095880443494400)]} 0 0
[junit4:junit4] 2> 353430 T2825 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[s!70 (1439095880446640128)]} 0 0
[junit4:junit4] 2> 353431 T2790 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[s!70 (1439095880446640128)]} 0 3
[junit4:junit4] 2> 353434 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[t!71 (1439095880451883008)]} 0 1
[junit4:junit4] 2> 353438 T2823 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[t!71 (1439095880455028736)]} 0 0
[junit4:junit4] 2> 353439 T2793 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[t!71 (1439095880455028736)]} 0 3
[junit4:junit4] 2> 353441 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[u!72 (1439095880460271616)]} 0 0
[junit4:junit4] 2> 353448 T2841 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[u!72 (1439095880465514496)]} 0 1
[junit4:junit4] 2> 353449 T2807 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[u!72 (1439095880465514496)]} 0 4
[junit4:junit4] 2> 353449 T2791 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[u!72]} 0 5
[junit4:junit4] 2> 353452 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[v!73 (1439095880471805952)]} 0 1
[junit4:junit4] 2> 353459 T2824 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[v!73 (1439095880476000256)]} 0 1
[junit4:junit4] 2> 353459 T2792 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[v!73 (1439095880476000256)]} 0 3
[junit4:junit4] 2> 353460 T2806 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[v!73]} 0 6
[junit4:junit4] 2> 353463 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[w!74 (1439095880482291712)]} 0 1
[junit4:junit4] 2> 353468 T2822 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[w!74 (1439095880485437440)]} 0 1
[junit4:junit4] 2> 353468 T2790 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[w!74 (1439095880485437440)]} 0 3
[junit4:junit4] 2> 353471 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[x!75 (1439095880491728896)]} 0 0
[junit4:junit4] 2> 353476 T2839 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[x!75 (1439095880494874624)]} 0 0
[junit4:junit4] 2> 353477 T2808 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[x!75 (1439095880494874624)]} 0 3
[junit4:junit4] 2> 353479 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[y!76 (1439095880500117504)]} 0 0
[junit4:junit4] 2> 353484 T2840 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[y!76 (1439095880503263232)]} 0 0
[junit4:junit4] 2> 353485 T2809 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[y!76 (1439095880503263232)]} 0 3
[junit4:junit4] 2> 353487 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[z!77 (1439095880508506112)]} 0 0
[junit4:junit4] 2> 353494 T2825 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[z!77 (1439095880513748992)]} 0 1
[junit4:junit4] 2> 353495 T2793 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[z!77 (1439095880513748992)]} 0 4
[junit4:junit4] 2> 353495 T2807 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[z!77]} 0 5
[junit4:junit4] 2> 353498 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[a!78 (1439095880520040448)]} 0 0
[junit4:junit4] 2> 353506 T2842 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[a!78 (1439095880525283328)]} 0 0
[junit4:junit4] 2> 353507 T2806 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[a!78 (1439095880525283328)]} 0 4
[junit4:junit4] 2> 353508 T2791 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[a!78]} 0 7
[junit4:junit4] 2> 353511 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[b!79 (1439095880532623360)]} 0 1
[junit4:junit4] 2> 353516 T2823 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[b!79 (1439095880535769088)]} 0 0
[junit4:junit4] 2> 353517 T2790 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[b!79 (1439095880535769088)]} 0 4
[junit4:junit4] 2> 353520 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[c!80 (1439095880543109120)]} 0 0
[junit4:junit4] 2> 353526 T2824 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[c!80 (1439095880546254848)]} 0 1
[junit4:junit4] 2> 353526 T2793 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[c!80 (1439095880546254848)]} 0 3
[junit4:junit4] 2> 353529 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[d!81 (1439095880552546304)]} 0 0
[junit4:junit4] 2> 353534 T2841 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[d!81 (1439095880555692032)]} 0 1
[junit4:junit4] 2> 353535 T2808 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[d!81 (1439095880555692032)]} 0 4
[junit4:junit4] 2> 353537 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[e!82 (1439095880560934912)]} 0 0
[junit4:junit4] 2> 353543 T2839 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[e!82 (1439095880564080640)]} 0 1
[junit4:junit4] 2> 353543 T2809 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[e!82 (1439095880564080640)]} 0 3
[junit4:junit4] 2> 353546 T2771 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[f!83 (1439095880570372096)]} 0 0
[junit4:junit4] 2> 353551 T2840 C785 P42486 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[f!83 (1439095880573517824)]} 0 0
[junit4:junit4] 2> 353552 T2807 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[f!83 (1439095880573517824)]} 0 3
[junit4:junit4] 2> 353554 T2772 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[g!84 (1439095880578760704)]} 0 0
[junit4:junit4] 2> 353559 T2825 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[g!84 (1439095880580857856)]} 0 1
[junit4:junit4] 2> 353559 T2791 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[g!84 (1439095880580857856)]} 0 3
[junit4:junit4] 2> 353562 T2773 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[h!85 (1439095880586100736)]} 0 1
[junit4:junit4] 2> 353568 T2824 C786 P40659 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:55944/wcv/n/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[h!85 (1439095880591343616)]} 0 0
[junit4:junit4] 2> 353569 T2792 C784 P55944 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={distrib.from=http://127.0.0.1:49506/wcv/n/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[h!85 (1439095880591343616)]} 0 3
[junit4:junit4] 2> 353570 T2808 C783 P49506 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[h!85]} 0 6
[junit4:junit4] 2> 353572 T2770 C782 P40525 oasup.LogUpdateProcessor.finish [collection1] webapp=/wcv/n path=/update params={wt=javabin&version=2} {add=[i!86 (1439095880597635072)]} 0 0
[junit4:junit4] 2> 353579 T2822 C786 P40659 oasup.LogUpdateProcessor.fi
[...truncated too long message...]
llection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:40525_wcv%2Fn",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:40525/wcv/n"}
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/election (5)
[junit4:junit4] 1> /solr/overseer_elect/election/89943491087433740-127.0.0.1:42486_wcv%2Fn-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89943491087433738-127.0.0.1:40659_wcv%2Fn-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89943491087433736-127.0.0.1:49506_wcv%2Fn-n_0000000002 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89943491087433734-127.0.0.1:55944_wcv%2Fn-n_0000000001 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89943491087433731-127.0.0.1:40525_wcv%2Fn-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89943491087433731-127.0.0.1:40525_wcv%2Fn-n_0000000000"}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 1> -
[junit4:junit4] 1> rstate:active live:true
[junit4:junit4] 1> rstate:active live:true
[junit4:junit4] 1> rstate:active live:true
[junit4:junit4] 1> rstate:active live:true
[junit4:junit4] 1> rstate:active live:true
[junit4:junit4] 1> rstate:active live:true
[junit4:junit4] 1> rstate:active live:true
[junit4:junit4] 1> rstate:active live:true
[junit4:junit4] 1> no one is recoverying
[junit4:junit4] 2> 484709 T2755 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 153321 T2754 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
[junit4:junit4] 2> 173354 T2754 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 1 thread leaked from SUITE scope at org.apache.solr.cloud.ShardSplitTest:
[junit4:junit4] 2> 1) Thread[id=2903, name=IPC Parameter Sending Thread #2, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
[junit4:junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] 2> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4:junit4] 2> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
[junit4:junit4] 2> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:360)
[junit4:junit4] 2> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:939)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 173354 T2754 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4:junit4] 2> 1) Thread[id=2903, name=IPC Parameter Sending Thread #2, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
[junit4:junit4] 2> 176356 T2754 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4:junit4] 2> 1) Thread[id=2903, name=IPC Parameter Sending Thread #2, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
[junit4:junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] 2> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4:junit4] 2> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
[junit4:junit4] 2> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:360)
[junit4:junit4] 2> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:939)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=is_IS, timezone=America/Argentina/Salta
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation 1.8.0-ea (32-bit)/cpus=8,threads=1,free=153235896,total=423866368
[junit4:junit4] 2> NOTE: All tests run in this JVM: [HdfsUnloadDistributedZkTest, BasicDistributedZk2Test, TestSchemaSimilarityResource, TestElisionMultitermQuery, TestSystemIdResolver, TestStressLucene, LeaderElectionIntegrationTest, HdfsCollectionsAPIDistributedZkTest, TestMergePolicyConfig, TestLMJelinekMercerSimilarityFactory, DateMathParserTest, TestRecoveryHdfs, TestManagedSchema, TestUniqueKeyFieldResource, TestQueryTypes, URLClassifyProcessorTest, OpenExchangeRatesOrgProviderTest, HdfsRecoveryZkTest, TestBM25SimilarityFactory, TestSolr4Spatial, SolrRequestParserTest, TestCloudManagedSchemaAddField, TestWordDelimiterFilterFactory, SpellPossibilityIteratorTest, HdfsSyncSliceTest, TestRandomFaceting, OutputWriterTest, DistanceFunctionTest, TestZkChroot, TestUpdate, JSONWriterTest, TestIndexSearcher, TestStressReorder, SpellCheckCollatorTest, SuggesterWFSTTest, TestReloadAndDeleteDocs, CollectionsAPIDistributedZkTest, ShardSplitTest]
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ShardSplitTest -Dtests.seed=575283B251B3D6A -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=is_IS -Dtests.timezone=America/Argentina/Salta -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 0.00s J1 | ShardSplitTest (suite) <<<
[junit4:junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.ShardSplitTest:
[junit4:junit4] > 1) Thread[id=2903, name=IPC Parameter Sending Thread #2, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
[junit4:junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4:junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
[junit4:junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:360)
[junit4:junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:939)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([575283B251B3D6A]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4] > 1) Thread[id=2903, name=IPC Parameter Sending Thread #2, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
[junit4:junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4:junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
[junit4:junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:360)
[junit4:junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:939)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([575283B251B3D6A]:0)
[junit4:junit4] Completed on J1 in 176.39s, 1 test, 2 errors <<< FAILURES!
[...truncated 586 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:190: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:443: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1245: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:889: There were test failures: 312 suites, 1316 tests, 2 suite-level errors, 655 ignored (3 assumptions)
Total time: 28 minutes 31 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-ea-b94 -client -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_25) - Build #
6350 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6350/
Java: 64bit/jdk1.7.0_25 -XX:+UseCompressedOops -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:35890/collection1lastClient and got 249 from http://127.0.0.1:49136/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:35890/collection1lastClient and got 249 from http://127.0.0.1:49136/collection1
at __randomizedtesting.SeedInfo.seed([BD3613953711EFA1:3CD09D8D404E8F9D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1045)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:237)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9605 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4] 2> 419074 T2118 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 419079 T2118 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-SyncSliceTest-1372423760296
[junit4:junit4] 2> 419079 T2118 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 419080 T2119 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 419180 T2118 oasc.ZkTestServer.run start zk server on port:50214
[junit4:junit4] 2> 419181 T2118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 419386 T2125 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f4781c8 name:ZooKeeperConnection Watcher:127.0.0.1:50214 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 419387 T2118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 419387 T2118 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 419391 T2118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 419393 T2127 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39d0ccfb name:ZooKeeperConnection Watcher:127.0.0.1:50214/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 419393 T2118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 419393 T2118 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 419395 T2118 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 419398 T2118 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 419399 T2118 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 419401 T2118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 419402 T2118 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 419406 T2118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 419406 T2118 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 419409 T2118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 419409 T2118 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 419411 T2118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 419412 T2118 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 419415 T2118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 419415 T2118 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 419418 T2118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 419419 T2118 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 419421 T2118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 419422 T2118 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 419424 T2118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 419425 T2118 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 419427 T2118 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 419427 T2118 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 419498 T2118 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 419500 T2118 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39279
[junit4:junit4] 2> 419500 T2118 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 419501 T2118 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 419501 T2118 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372423760648
[junit4:junit4] 2> 419501 T2118 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372423760648/solr.xml
[junit4:junit4] 2> 419502 T2118 oasc.CoreContainer.<init> New CoreContainer 1016123142
[junit4:junit4] 2> 419502 T2118 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372423760648/'
[junit4:junit4] 2> 419502 T2118 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372423760648/'
[junit4:junit4] 2> 419570 T2118 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 419571 T2118 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 419571 T2118 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 419572 T2118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 419572 T2118 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 419572 T2118 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 419573 T2118 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 419573 T2118 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 419573 T2118 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 419573 T2118 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 419577 T2118 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 419578 T2118 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50214/solr
[junit4:junit4] 2> 419578 T2118 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 419579 T2118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 419580 T2138 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ba872cb name:ZooKeeperConnection Watcher:127.0.0.1:50214 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 419581 T2118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 419583 T2118 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 419586 T2118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 419587 T2140 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50f4eda2 name:ZooKeeperConnection Watcher:127.0.0.1:50214/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 419587 T2118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 419589 T2118 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 419592 T2118 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 419595 T2118 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 419596 T2118 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39279_
[junit4:junit4] 2> 419597 T2118 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39279_
[junit4:junit4] 2> 419599 T2118 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 419602 T2118 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 419604 T2118 oasc.Overseer.start Overseer (id=89943163554889731-127.0.0.1:39279_-n_0000000000) starting
[junit4:junit4] 2> 419610 T2118 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 419614 T2142 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 419614 T2118 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 419616 T2118 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 419617 T2118 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 419619 T2141 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 419622 T2143 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 419622 T2143 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 419623 T2143 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 421123 T2141 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 421124 T2141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39279_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39279"}
[junit4:junit4] 2> 421124 T2141 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 421124 T2141 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 421127 T2140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 421623 T2143 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 421624 T2143 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372423760648/collection1
[junit4:junit4] 2> 421624 T2143 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 421625 T2143 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 421625 T2143 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 421626 T2143 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372423760648/collection1/'
[junit4:junit4] 2> 421627 T2143 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-controljetty-1372423760648/collection1/lib/README' to classloader
[junit4:junit4] 2> 421627 T2143 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-controljetty-1372423760648/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 421652 T2143 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 421683 T2143 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 421684 T2143 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 421688 T2143 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 422109 T2143 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 422132 T2143 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 422134 T2143 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 422146 T2143 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 422149 T2143 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 422152 T2143 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 422153 T2143 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 422153 T2143 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 422153 T2143 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 422154 T2143 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 422154 T2143 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 422155 T2143 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 422155 T2143 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372423760648/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372423760296/control/data/
[junit4:junit4] 2> 422155 T2143 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aff2be8
[junit4:junit4] 2> 422156 T2143 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372423760296/control/data
[junit4:junit4] 2> 422156 T2143 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372423760296/control/data/index/
[junit4:junit4] 2> 422156 T2143 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372423760296/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 422156 T2143 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372423760296/control/data/index
[junit4:junit4] 2> 422158 T2143 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@568ab368 lockFactory=org.apache.lucene.store.NativeFSLockFactory@cd828e0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 422158 T2143 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 422160 T2143 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 422160 T2143 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 422161 T2143 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 422161 T2143 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 422162 T2143 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 422162 T2143 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 422162 T2143 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 422163 T2143 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 422163 T2143 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 422165 T2143 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 422168 T2143 oass.SolrIndexSearcher.<init> Opening Searcher@3c2464c9 main
[junit4:junit4] 2> 422168 T2143 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 422168 T2143 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 422172 T2144 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3c2464c9 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 422174 T2143 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 422174 T2143 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39279 collection:control_collection shard:shard1
[junit4:junit4] 2> 422175 T2143 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 422180 T2143 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4] 2> 422182 T2143 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 422182 T2143 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 422182 T2143 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39279/collection1/
[junit4:junit4] 2> 422182 T2143 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 422182 T2143 oasc.SyncStrategy.syncToMe http://127.0.0.1:39279/collection1/ has no replicas
[junit4:junit4] 2> 422182 T2143 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39279/collection1/ shard1
[junit4:junit4] 2> 422183 T2143 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 422631 T2141 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 422637 T2140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 422688 T2143 oasc.ZkController.register We are http://127.0.0.1:39279/collection1/ and leader is http://127.0.0.1:39279/collection1/
[junit4:junit4] 2> 422689 T2143 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39279
[junit4:junit4] 2> 422689 T2143 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 422689 T2143 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 422689 T2143 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 422691 T2143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 422692 T2118 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 422692 T2118 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 422693 T2118 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 422696 T2118 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 422697 T2118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 422698 T2147 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7bfc1e8a name:ZooKeeperConnection Watcher:127.0.0.1:50214/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 422698 T2118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 422700 T2118 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 422702 T2118 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 422762 T2118 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 422764 T2118 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35890
[junit4:junit4] 2> 422764 T2118 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 422764 T2118 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 422765 T2118 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372423763919
[junit4:junit4] 2> 422765 T2118 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372423763919/solr.xml
[junit4:junit4] 2> 422765 T2118 oasc.CoreContainer.<init> New CoreContainer 1465730992
[junit4:junit4] 2> 422766 T2118 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372423763919/'
[junit4:junit4] 2> 422766 T2118 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372423763919/'
[junit4:junit4] 2> 422809 T2118 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 422810 T2118 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 422810 T2118 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 422810 T2118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 422810 T2118 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 422811 T2118 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 422811 T2118 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 422811 T2118 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 422811 T2118 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 422812 T2118 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 422815 T2118 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 422815 T2118 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50214/solr
[junit4:junit4] 2> 422815 T2118 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 422816 T2118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 422818 T2158 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34886f76 name:ZooKeeperConnection Watcher:127.0.0.1:50214 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 422818 T2118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 422820 T2118 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 422823 T2118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 422824 T2160 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a9ca132 name:ZooKeeperConnection Watcher:127.0.0.1:50214/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 422824 T2118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 422828 T2118 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 423831 T2118 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35890_
[junit4:junit4] 2> 423832 T2118 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35890_
[junit4:junit4] 2> 423835 T2140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 423836 T2147 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 423836 T2160 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 423837 T2140 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 423841 T2161 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 423841 T2161 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 423842 T2161 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 424141 T2141 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 424142 T2141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:39279_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39279"}
[junit4:junit4] 2> 424145 T2141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35890_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35890"}
[junit4:junit4] 2> 424145 T2141 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 424145 T2141 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 424149 T2147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 424149 T2140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 424149 T2160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 424842 T2161 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 424843 T2161 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372423763919/collection1
[junit4:junit4] 2> 424843 T2161 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 424844 T2161 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 424844 T2161 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 424852 T2161 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372423763919/collection1/'
[junit4:junit4] 2> 424852 T2161 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty1-1372423763919/collection1/lib/README' to classloader
[junit4:junit4] 2> 424853 T2161 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty1-1372423763919/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 424896 T2161 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 424940 T2161 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 424941 T2161 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 424947 T2161 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 425519 T2161 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 425525 T2161 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 425528 T2161 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 425540 T2161 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 425544 T2161 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 425548 T2161 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 425549 T2161 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 425550 T2161 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 425550 T2161 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 425551 T2161 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 425551 T2161 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 425552 T2161 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 425552 T2161 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372423763919/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty1/
[junit4:junit4] 2> 425552 T2161 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aff2be8
[junit4:junit4] 2> 425553 T2161 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty1
[junit4:junit4] 2> 425554 T2161 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty1/index/
[junit4:junit4] 2> 425554 T2161 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 425554 T2161 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty1/index
[junit4:junit4] 2> 425556 T2161 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6aea1def lockFactory=org.apache.lucene.store.NativeFSLockFactory@4331f521),segFN=segments_1,generation=1}
[junit4:junit4] 2> 425557 T2161 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 425560 T2161 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 425560 T2161 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 425561 T2161 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 425562 T2161 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 425563 T2161 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 425563 T2161 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 425564 T2161 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 425564 T2161 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 425565 T2161 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 425569 T2161 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 425572 T2161 oass.SolrIndexSearcher.<init> Opening Searcher@71127443 main
[junit4:junit4] 2> 425573 T2161 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 425574 T2161 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 425581 T2162 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@71127443 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 425584 T2161 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 425585 T2161 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35890 collection:collection1 shard:shard1
[junit4:junit4] 2> 425586 T2161 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 425594 T2161 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4] 2> 425596 T2161 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 425596 T2161 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 425597 T2161 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35890/collection1/
[junit4:junit4] 2> 425597 T2161 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 425597 T2161 oasc.SyncStrategy.syncToMe http://127.0.0.1:35890/collection1/ has no replicas
[junit4:junit4] 2> 425597 T2161 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35890/collection1/ shard1
[junit4:junit4] 2> 425598 T2161 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 425653 T2141 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 425659 T2140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 425659 T2160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 425659 T2147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 425708 T2161 oasc.ZkController.register We are http://127.0.0.1:35890/collection1/ and leader is http://127.0.0.1:35890/collection1/
[junit4:junit4] 2> 425708 T2161 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35890
[junit4:junit4] 2> 425708 T2161 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 425709 T2161 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 425709 T2161 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 425710 T2161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 425711 T2118 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 425712 T2118 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 425712 T2118 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 425786 T2118 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 425788 T2118 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44717
[junit4:junit4] 2> 425789 T2118 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 425789 T2118 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 425789 T2118 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372423766933
[junit4:junit4] 2> 425790 T2118 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372423766933/solr.xml
[junit4:junit4] 2> 425790 T2118 oasc.CoreContainer.<init> New CoreContainer 788102314
[junit4:junit4] 2> 425791 T2118 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372423766933/'
[junit4:junit4] 2> 425791 T2118 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372423766933/'
[junit4:junit4] 2> 425844 T2118 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 425844 T2118 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 425845 T2118 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 425845 T2118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 425845 T2118 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 425846 T2118 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 425846 T2118 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 425846 T2118 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 425847 T2118 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 425847 T2118 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 425851 T2118 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 425852 T2118 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50214/solr
[junit4:junit4] 2> 425852 T2118 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 425853 T2118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 425854 T2174 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4334b104 name:ZooKeeperConnection Watcher:127.0.0.1:50214 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 425855 T2118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 425857 T2118 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 425861 T2118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 425862 T2176 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6662e046 name:ZooKeeperConnection Watcher:127.0.0.1:50214/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 425862 T2118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 425866 T2118 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 426869 T2118 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44717_
[junit4:junit4] 2> 426870 T2118 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44717_
[junit4:junit4] 2> 426872 T2140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 426874 T2147 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 426874 T2176 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 426874 T2160 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 426874 T2147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 426874 T2160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 426875 T2140 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 426879 T2177 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 426879 T2177 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 426880 T2177 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 427163 T2141 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 427164 T2141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35890_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35890"}
[junit4:junit4] 2> 427167 T2141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44717_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44717"}
[junit4:junit4] 2> 427167 T2141 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 427167 T2141 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 427169 T2140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 427169 T2147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 427170 T2160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 427170 T2176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 427880 T2177 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 427881 T2177 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372423766933/collection1
[junit4:junit4] 2> 427881 T2177 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 427882 T2177 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 427882 T2177 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 427883 T2177 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372423766933/collection1/'
[junit4:junit4] 2> 427884 T2177 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty2-1372423766933/collection1/lib/README' to classloader
[junit4:junit4] 2> 427884 T2177 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty2-1372423766933/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 427909 T2177 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 427938 T2177 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 427939 T2177 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 427943 T2177 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 428249 T2177 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 428254 T2177 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 428256 T2177 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 428266 T2177 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 428269 T2177 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 428271 T2177 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 428272 T2177 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 428272 T2177 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 428273 T2177 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 428273 T2177 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 428274 T2177 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 428274 T2177 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 428274 T2177 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372423766933/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty2/
[junit4:junit4] 2> 428274 T2177 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aff2be8
[junit4:junit4] 2> 428275 T2177 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty2
[junit4:junit4] 2> 428275 T2177 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty2/index/
[junit4:junit4] 2> 428275 T2177 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 428276 T2177 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty2/index
[junit4:junit4] 2> 428277 T2177 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4ac221b4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d5c615b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 428277 T2177 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 428279 T2177 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 428279 T2177 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 428280 T2177 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 428280 T2177 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 428281 T2177 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 428281 T2177 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 428281 T2177 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 428282 T2177 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 428282 T2177 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 428284 T2177 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 428287 T2177 oass.SolrIndexSearcher.<init> Opening Searcher@3a269f5c main
[junit4:junit4] 2> 428287 T2177 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 428287 T2177 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 428292 T2178 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3a269f5c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 428294 T2177 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 428295 T2177 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44717 collection:collection1 shard:shard1
[junit4:junit4] 2> 428297 T2177 oasc.ZkController.register We are http://127.0.0.1:44717/collection1/ and leader is http://127.0.0.1:35890/collection1/
[junit4:junit4] 2> 428297 T2177 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44717
[junit4:junit4] 2> 428298 T2177 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 428298 T2177 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C452 name=collection1 org.apache.solr.core.SolrCore@fe2a8a5 url=http://127.0.0.1:44717/collection1 node=127.0.0.1:44717_ C452_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:44717_, base_url=http://127.0.0.1:44717}
[junit4:junit4] 2> 428298 T2179 C452 P44717 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 428299 T2179 C452 P44717 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 428299 T2177 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 428299 T2179 C452 P44717 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 428299 T2179 C452 P44717 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 428300 T2118 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 428300 T2118 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 428300 T2179 C452 P44717 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 428301 T2118 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 428306 T2156 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 428362 T2118 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 428365 T2118 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49136
[junit4:junit4] 2> 428365 T2118 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 428365 T2118 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 428366 T2118 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372423769522
[junit4:junit4] 2> 428366 T2118 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372423769522/solr.xml
[junit4:junit4] 2> 428366 T2118 oasc.CoreContainer.<init> New CoreContainer 672546756
[junit4:junit4] 2> 428367 T2118 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372423769522/'
[junit4:junit4] 2> 428367 T2118 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372423769522/'
[junit4:junit4] 2> 428428 T2118 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 428429 T2118 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 428429 T2118 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 428429 T2118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 428430 T2118 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 428430 T2118 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 428430 T2118 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 428430 T2118 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 428431 T2118 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 428431 T2118 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 428434 T2118 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 428435 T2118 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50214/solr
[junit4:junit4] 2> 428435 T2118 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 428436 T2118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 428437 T2191 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b869c8d name:ZooKeeperConnection Watcher:127.0.0.1:50214 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 428437 T2118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 428439 T2118 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 428442 T2118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 428443 T2193 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@645de8f2 name:ZooKeeperConnection Watcher:127.0.0.1:50214/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 428443 T2118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 428446 T2118 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 428674 T2141 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 428675 T2141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44717_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44717"}
[junit4:junit4] 2> 428678 T2140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 428679 T2193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 428679 T2160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 428679 T2147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 428679 T2176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 429307 T2156 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 429307 T2156 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node2&state=recovering&nodeName=127.0.0.1:44717_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 429449 T2118 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49136_
[junit4:junit4] 2> 429450 T2118 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49136_
[junit4:junit4] 2> 429453 T2176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 429453 T2140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 429454 T2160 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 429454 T2160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 429455 T2147 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 429455 T2147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 429455 T2193 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 429455 T2193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 429456 T2176 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 429457 T2140 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 429465 T2194 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 429465 T2194 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 429466 T2194 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 430183 T2141 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 430184 T2141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49136_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49136"}
[junit4:junit4] 2> 430184 T2141 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 430184 T2141 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 430188 T2140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 430189 T2176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 430189 T2147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 430188 T2160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 430191 T2193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 430467 T2194 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 430467 T2194 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372423769522/collection1
[junit4:junit4] 2> 430467 T2194 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 430468 T2194 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 430469 T2194 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 430470 T2194 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372423769522/collection1/'
[junit4:junit4] 2> 430471 T2194 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty3-1372423769522/collection1/lib/README' to classloader
[junit4:junit4] 2> 430471 T2194 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty3-1372423769522/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 430520 T2194 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 430566 T2194 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 430574 T2194 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 430589 T2194 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 430952 T2194 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 430957 T2194 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 430959 T2194 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 430969 T2194 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 430972 T2194 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 430974 T2194 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 430975 T2194 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 430976 T2194 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 430976 T2194 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 430977 T2194 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 430977 T2194 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 430977 T2194 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 430977 T2194 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372423769522/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty3/
[junit4:junit4] 2> 430977 T2194 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aff2be8
[junit4:junit4] 2> 430978 T2194 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty3
[junit4:junit4] 2> 430978 T2194 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty3/index/
[junit4:junit4] 2> 430979 T2194 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 430979 T2194 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty3/index
[junit4:junit4] 2> 430980 T2194 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@75e0b7cf lockFactory=org.apache.lucene.store.NativeFSLockFactory@450592c7),segFN=segments_1,generation=1}
[junit4:junit4] 2> 430980 T2194 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 430982 T2194 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 430983 T2194 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 430983 T2194 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 430984 T2194 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 430984 T2194 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 430984 T2194 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 430985 T2194 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 430985 T2194 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 430986 T2194 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 430988 T2194 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 430990 T2194 oass.SolrIndexSearcher.<init> Opening Searcher@36d96b3c main
[junit4:junit4] 2> 430991 T2194 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 430991 T2194 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 430995 T2195 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@36d96b3c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 430997 T2194 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 430997 T2194 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49136 collection:collection1 shard:shard1
[junit4:junit4] 2> 431000 T2194 oasc.ZkController.register We are http://127.0.0.1:49136/collection1/ and leader is http://127.0.0.1:35890/collection1/
[junit4:junit4] 2> 431000 T2194 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49136
[junit4:junit4] 2> 431000 T2194 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 431000 T2194 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C453 name=collection1 org.apache.solr.core.SolrCore@7595fd83 url=http://127.0.0.1:49136/collection1 node=127.0.0.1:49136_ C453_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:49136_, base_url=http://127.0.0.1:49136}
[junit4:junit4] 2> 431001 T2196 C453 P49136 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 431001 T2194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 431001 T2196 C453 P49136 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 431002 T2196 C453 P49136 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 431002 T2196 C453 P49136 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 431002 T2118 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 431002 T2118 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 431003 T2118 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 431005 T2196 C453 P49136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 431010 T2156 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 431064 T2118 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 431066 T2118 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47359
[junit4:junit4] 2> 431066 T2118 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 431067 T2118 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 431067 T2118 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372423772224
[junit4:junit4] 2> 431067 T2118 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372423772224/solr.xml
[junit4:junit4] 2> 431068 T2118 oasc.CoreContainer.<init> New CoreContainer 544465729
[junit4:junit4] 2> 431068 T2118 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372423772224/'
[junit4:junit4] 2> 431068 T2118 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372423772224/'
[junit4:junit4] 2> 431112 T2118 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 431113 T2118 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 431113 T2118 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 431113 T2118 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 431114 T2118 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 431114 T2118 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 431114 T2118 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 431115 T2118 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 431115 T2118 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 431115 T2118 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 431118 T2118 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 431119 T2118 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50214/solr
[junit4:junit4] 2> 431119 T2118 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 431120 T2118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 431122 T2208 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31f1b185 name:ZooKeeperConnection Watcher:127.0.0.1:50214 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 431122 T2118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 431123 T2118 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 431127 T2118 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 431128 T2210 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a5fcf47 name:ZooKeeperConnection Watcher:127.0.0.1:50214/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 431128 T2118 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 431132 T2118 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C454 name=collection1 org.apache.solr.core.SolrCore@fe2a8a5 url=http://127.0.0.1:44717/collection1 node=127.0.0.1:44717_ C454_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:44717_, base_url=http://127.0.0.1:44717}
[junit4:junit4] 2> 431308 T2179 C454 P44717 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:35890/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 431309 T2179 C454 P44717 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44717 START replicas=[http://127.0.0.1:35890/collection1/] nUpdates=100
[junit4:junit4] 2> 431309 T2179 C454 P44717 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 431310 T2179 C454 P44717 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 431310 T2179 C454 P44717 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 431310 T2179 C454 P44717 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 431310 T2179 C454 P44717 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 431311 T2179 C454 P44717 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:35890/collection1/. core=collection1
[junit4:junit4] 2> 431311 T2179 C454 P44717 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C455 name=collection1 org.apache.solr.core.SolrCore@17ac4e1a url=http://127.0.0.1:35890/collection1 node=127.0.0.1:35890_ C455_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:35890_, base_url=http://127.0.0.1:35890, leader=true}
[junit4:junit4] 2> 431315 T2154 C455 P35890 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 431317 T2154 C455 P35890 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 431318 T2154 C455 P35890 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6aea1def lockFactory=org.apache.lucene.store.NativeFSLockFactory@4331f521),segFN=segments_1,generation=1}
[junit4:junit4] 2> 431318 T2154 C455 P35890 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 431319 T2154 C455 P35890 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 431319 T2154 C455 P35890 oass.SolrIndexSearcher.<init> Opening Searcher@5a0ab07c realtime
[junit4:junit4] 2> 431319 T2154 C455 P35890 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 431320 T2154 C455 P35890 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4] 2> 431321 T2179 C454 P44717 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 431321 T2179 C454 P44717 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 431322 T2155 C455 P35890 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 431322 T2155 C455 P35890 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 431323 T2179 C454 P44717 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 431323 T2179 C454 P44717 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 431323 T2179 C454 P44717 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 431323 T2179 C454 P44717 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 431324 T2179 C454 P44717 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 431694 T2141 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 431695 T2141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49136_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49136"}
[junit4:junit4] 2> 431698 T2141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44717_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44717"}
[junit4:junit4] 2> 431701 T2140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 431701 T2160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 431701 T2193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 431701 T2210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 431701 T2147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 431701 T2176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 432011 T2156 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 432011 T2156 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node3&state=recovering&nodeName=127.0.0.1:49136_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 432134 T2118 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47359_
[junit4:junit4] 2> 432136 T2118 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47359_
[junit4:junit4] 2> 432138 T2140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 432138 T2176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 432138 T2210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 432139 T2147 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 432139 T2160 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 432139 T2147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 432138 T2193 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 432139 T2160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 432139 T2193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 432140 T2140 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 432141 T2176 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 432141 T2210 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 432145 T2212 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 432145 T2212 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 432146 T2212 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 433207 T2141 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 433207 T2141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:47359_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47359"}
[junit4:junit4] 2> 433208 T2141 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 433208 T2141 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 433211 T2140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 433211 T2193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 433212 T2147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 433212 T2210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 433212 T2160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 433212 T2176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> C453_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:49136_, base_url=http://127.0.0.1:49136}
[junit4:junit4] 2> 434012 T2196 C453 P49136 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:35890/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 434012 T2196 C453 P49136 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:49136 START replicas=[http://127.0.0.1:35890/collection1/] nUpdates=100
[junit4:junit4] 2> 434012 T2196 C453 P49136 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 434013 T2196 C453 P49136 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 434013 T2196 C453 P49136 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 434014 T2196 C453 P49136 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 434014 T2196 C453 P49136 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 434014 T2196 C453 P49136 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:35890/collection1/. core=collection1
[junit4:junit4] 2> 434014 T2196 C453 P49136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 434015 T2156 C455 P35890 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 434017 T2155 C455 P35890 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 434018 T2155 C455 P35890 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 434018 T2155 C455 P35890 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 434018 T2155 C455 P35890 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 1
[junit4:junit4] 2> 434019 T2196 C453 P49136 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 434019 T2196 C453 P49136 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 434021 T2153 C455 P35890 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 434021 T2196 C453 P49136 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 434022 T2196 C453 P49136 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 434022 T2196 C453 P49136 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 434022 T2196 C453 P49136 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 434023 T2196 C453 P49136 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 434147 T2212 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 434147 T2212 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372423772224/collection1
[junit4:junit4] 2> 434147 T2212 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 434148 T2212 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 434148 T2212 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 434150 T2212 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372423772224/collection1/'
[junit4:junit4] 2> 434150 T2212 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty4-1372423772224/collection1/lib/README' to classloader
[junit4:junit4] 2> 434151 T2212 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.SyncSliceTest-jetty4-1372423772224/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 434182 T2212 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 434215 T2212 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 434217 T2212 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 434220 T2212 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 434584 T2212 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 434589 T2212 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 434592 T2212 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 434602 T2212 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 434605 T2212 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 434608 T2212 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 434609 T2212 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 434610 T2212 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 434610 T2212 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 434611 T2212 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 434611 T2212 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 434611 T2212 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 434612 T2212 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372423772224/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty4/
[junit4:junit4] 2> 434612 T2212 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aff2be8
[junit4:junit4] 2> 434613 T2212 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty4
[junit4:junit4] 2> 434613 T2212 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty4/index/
[junit4:junit4] 2> 434613 T2212 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 434613 T2212 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty4/index
[junit4:junit4] 2> 434615 T2212 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1cf0d8b4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@584aca85),segFN=segments_1,generation=1}
[junit4:junit4] 2> 434615 T2212 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 434617 T2212 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 434617 T2212 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 434618 T2212 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 434618 T2212 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 434619 T2212 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 434619 T2212 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 434620 T2212 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 434621 T2212 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 434621 T2212 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 434624 T2212 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 434626 T2212 oass.SolrIndexSearcher.<init> Opening Searcher@35fd37f5 main
[junit4:junit4] 2> 434627 T2212 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 434627 T2212 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 434633 T2214 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@35fd37f5 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 434635 T2212 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 434635 T2212 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47359 collection:collection1 shard:shard1
[junit4:junit4] 2> 434638 T2212 oasc.ZkController.register We are http://127.0.0.1:47359/collection1/ and leader is http://127.0.0.1:35890/collection1/
[junit4:junit4] 2> 434638 T2212 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47359
[junit4:junit4] 2> 434638 T2212 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 434639 T2212 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C456 name=collection1 org.apache.solr.core.SolrCore@2519d039 url=http://127.0.0.1:47359/collection1 node=127.0.0.1:47359_ C456_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:47359_, base_url=http://127.0.0.1:47359}
[junit4:junit4] 2> 434639 T2215 C456 P47359 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 434639 T2215 C456 P47359 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 434639 T2212 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 434640 T2215 C456 P47359 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 434640 T2215 C456 P47359 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 434640 T2118 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 434641 T2118 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 434641 T2118 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 434644 T2215 C456 P47359 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 434645 T2118 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 434646 T2118 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30 for each attempt
[junit4:junit4] 2> 434647 T2118 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
[junit4:junit4] 2> 434647 T2118 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 434648 T2154 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 434715 T2141 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 434716 T2141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49136_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49136"}
[junit4:junit4] 2> 434719 T2141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:47359_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47359"}
[junit4:junit4] 2> 434723 T2140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 434723 T2176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 434723 T2210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 434723 T2160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 434723 T2147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 434723 T2193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 435649 T2154 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 435649 T2154 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node4&state=recovering&nodeName=127.0.0.1:47359_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 435649 T2118 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 436651 T2118 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C456_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:47359_, base_url=http://127.0.0.1:47359}
[junit4:junit4] 2> 437650 T2215 C456 P47359 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:35890/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 437650 T2215 C456 P47359 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:47359 START replicas=[http://127.0.0.1:35890/collection1/] nUpdates=100
[junit4:junit4] 2> 437651 T2215 C456 P47359 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 437651 T2215 C456 P47359 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 437651 T2215 C456 P47359 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 437651 T2215 C456 P47359 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 437651 T2215 C456 P47359 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 437651 T2215 C456 P47359 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:35890/collection1/. core=collection1
[junit4:junit4] 2> 437652 T2215 C456 P47359 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C457 name=collection1 org.apache.solr.core.SolrCore@17ac4e1a url=http://127.0.0.1:35890/collection1 node=127.0.0.1:35890_ C457_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:35890_, base_url=http://127.0.0.1:35890, leader=true}
[junit4:junit4] 2> 437653 T2154 C457 P35890 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 437653 T2118 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 437657 T2153 C457 P35890 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 437657 T2153 C457 P35890 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 437658 T2153 C457 P35890 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 437658 T2153 C457 P35890 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 1
[junit4:junit4] 2> 437658 T2215 C456 P47359 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 437659 T2215 C456 P47359 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 437661 T2156 C457 P35890 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 437661 T2215 C456 P47359 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 437661 T2215 C456 P47359 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 437662 T2215 C456 P47359 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 437662 T2215 C456 P47359 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 437663 T2215 C456 P47359 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 437730 T2141 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 437731 T2141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:47359_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47359"}
[junit4:junit4] 2> 437734 T2140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 437734 T2160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 437735 T2176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 437734 T2147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 437734 T2193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 437735 T2210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 438655 T2118 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 438656 T2118 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C458 name=collection1 org.apache.solr.core.SolrCore@77713854 url=http://127.0.0.1:39279/collection1 node=127.0.0.1:39279_ C458_STATE=coll:control_collection core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:39279_, base_url=http://127.0.0.1:39279, leader=true}
[junit4:junit4] 2> 438661 T2134 C458 P39279 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 438663 T2134 C458 P39279 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@568ab368 lockFactory=org.apache.lucene.store.NativeFSLockFactory@cd828e0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 438663 T2134 C458 P39279 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 438664 T2134 C458 P39279 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 438664 T2134 C458 P39279 oass.SolrIndexSearcher.<init> Opening Searcher@5d967118 main
[junit4:junit4] 2> 438664 T2134 C458 P39279 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 438665 T2144 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5d967118 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 438666 T2134 C458 P39279 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> ASYNC NEW_CORE C459 name=collection1 org.apache.solr.core.SolrCore@fe2a8a5 url=http://127.0.0.1:44717/collection1 node=127.0.0.1:44717_ C459_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:44717_, base_url=http://127.0.0.1:44717}
[junit4:junit4] 2> 438672 T2169 C459 P44717 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:35890/collection1/, StdNode: http://127.0.0.1:44717/collection1/, StdNode: http://127.0.0.1:49136/collection1/, StdNode: http://127.0.0.1:47359/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 438675 T2171 C459 P44717 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 438675 T2156 C457 P35890 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 438677 T2156 C457 P35890 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 438678 T2156 C457 P35890 oass.SolrIndexSearcher.<init> Opening Searcher@3eb906b4 main
[junit4:junit4] 2> 438678 T2156 C457 P35890 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 438678 T2171 C459 P44717 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4ac221b4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d5c615b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 438679 T2171 C459 P44717 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 438680 T2171 C459 P44717 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 438680 T2162 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3eb906b4 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 438680 T2171 C459 P44717 oass.SolrIndexSearcher.<init> Opening Searcher@5e698704 main
[junit4:junit4] 2> C456_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:47359_, base_url=http://127.0.0.1:47359}
[junit4:junit4] 2> 438680 T2204 C456 P47359 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C460 name=collection1 org.apache.solr.core.SolrCore@7595fd83 url=http://127.0.0.1:49136/collection1 node=127.0.0.1:49136_ C460_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:49136_, base_url=http://127.0.0.1:49136}
[junit4:junit4] 2> 438680 T2189 C460 P49136 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 438681 T2171 C459 P44717 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 438681 T2156 C457 P35890 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 438682 T2178 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5e698704 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 438682 T2204 C456 P47359 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1cf0d8b4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@584aca85),segFN=segments_1,generation=1}
[junit4:junit4] 2> 438683 T2189 C460 P49136 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@75e0b7cf lockFactory=org.apache.lucene.store.NativeFSLockFactory@450592c7),segFN=segments_1,generation=1}
[junit4:junit4] 2> 438683 T2171 C459 P44717 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 438683 T2204 C456 P47359 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 438683 T2189 C460 P49136 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 438684 T2204 C456 P47359 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 438684 T2189 C460 P49136 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 438685 T2204 C456 P47359 oass.SolrIndexSearcher.<init> Opening Searcher@377ce890 main
[junit4:junit4] 2> 438685 T2189 C460 P49136 oass.SolrIndexSearcher.<init> Opening Searcher@78f40b03 main
[junit4:junit4] 2> 438686 T2204 C456 P47359 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 438686 T2189 C460 P49136 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 438686 T2195 oasc.JmxMonitoredMap.put WARN Failed to register info bean: searcher javax.management.InstanceAlreadyExistsException: solr/collection1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4] 2> at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:305)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1844)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1721)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2>
[junit4:junit4] 2> 438687 T2214 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@377ce890 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 438687 T2195 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@78f40b03 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 438688 T2204 C456 P47359 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 438688 T2189 C460 P49136 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 438688 T2169 C459 P44717 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 16
[junit4:junit4] 2> 438689 T2118 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 438691 T2153 C457 P35890 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 438693 T2172 C459 P44717 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 438695 T2187 C460 P49136 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 438696 T2206 C456 P47359 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 440701 T2136 C458 P39279 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1439090639546351616)} 0 2
[junit4:junit4] 2> 440707 T2172 C459 P44717 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1439090639551594496&update.from=http://127.0.0.1:35890/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439090639551594496)} 0 1
[junit4:junit4] 2> 440707 T2204 C456 P47359 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1439090639551594496&update.from=http://127.0.0.1:35890/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439090639551594496)} 0 1
[junit4:junit4] 2> 440707 T2189 C460 P49136 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1439090639551594496&update.from=http://127.0.0.1:35890/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439090639551594496)} 0 1
[junit4:junit4] 2> 440708 T2155 C457 P35890 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1439090639551594496)} 0 4
[junit4:junit4] 2> 440708 T2186 C460 P49136 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 6
[junit4:junit4] 2> 440712 T2134 C458 P39279 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1439090639558934528)]} 0 2
[junit4:junit4] 2> 440720 T2187 C460 P49136 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:35890/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439090639564177408)]} 0 1
[junit4:junit4] 2> 440721 T2171 C459 P44717 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:35890/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439090639564177408)]} 0 2
[junit4:junit4] 2> 440720 T2205 C456 P47359 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:35890/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439090639564177408)]} 0 1
[junit4:junit4] 2> 440722 T2153 C457 P35890 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:44717/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1439090639564177408)]} 0 6
[junit4:junit4] 2> 440722 T2170 C459 P44717 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0]} 0 8
[junit4:junit4] 2> 440725 T2133 C458 P39279 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1439090639572566016)]} 0 1
[junit4:junit4] 2> 440733 T2172 C459 P44717 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:35890/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439090639577808896)]} 0 1
[junit4:junit4] 2> 440733 T2206 C456 P47359 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:35890/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439090639577808896)]} 0 1
[junit4:junit4] 2> 440733 T2188 C460 P49136 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:35890/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439090639577808896)]} 0 1
[junit4:junit4] 2> 440734 T2154 C457 P35890 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:44717/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1439090639577808896)]} 0 5
[junit4:junit4] 2> 440734 T2169 C459 P44717 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1]} 0 7
[junit4:junit4] 2> 440737 T2135 C458 P39279 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1439090639586197504)]} 0 1
[junit4:junit4] 2> 440741 T2156 C457 P35890 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:44717/collection1/]
[junit4:junit4] 2> 440742 T2156 C457 P35890 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44717/collection1/ against:[http://127.0.0.1:44717/collection1/] result:true
[junit4:junit4] 2> 440742 T2156 C457 P35890 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:49136/collection1/ against:[http://127.0.0.1:44717/collection1/] result:false
[junit4:junit4] 2> 440742 T2156 C457 P35890 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:47359/collection1/ against:[http://127.0.0.1:44717/collection1/] result:false
[junit4:junit4] 2> 440746 T2186 C460 P49136 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:35890/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:44717/collection1/&wt=javabin&version=2} {add=[2 (1439090639592488960)]} 0 1
[junit4:junit4] 2> 440746 T2203 C456 P47359 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:35890/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:44717/collection1/&wt=javabin&version=2} {add=[2 (1439090639592488960)]} 0 1
[junit4:junit4] 2> 440747 T2156 C457 P35890 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49136/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:44717/collection1/&wt=javabin&version=2} {add=[2 (1439090639592488960)]} 0 6
[junit4:junit4] 2> 440747 T2189 C460 P49136 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:44717/collection1/&wt=javabin&version=2} {add=[2]} 0 8
[junit4:junit4] 2> 440750 T2136 C458 P39279 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1439090639598780416)]} 0 1
[junit4:junit4] 2> 440752 T2153 C457 P35890 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:49136/collection1/, http://127.0.0.1:44717/collection1/]
[junit4:junit4] 2> 440752 T2153 C457 P35890 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44717/collection1/ against:[http://127.0.0.1:49136/collection1/, http://127.0.0.1:44717/collection1/] result:true
[junit4:junit4] 2> 440752 T2153 C457 P35890 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:49136/collection1/ against:[http://127.0.0.1:49136/collection1/, http://127.0.0.1:44717/collection1/] result:true
[junit4:junit4] 2> 440753 T2153 C457 P35890 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:47359/collection1/ against:[http://127.0.0.1:49136/collection1/, http://127.0.0.1:44717/collection1/] result:false
[junit4:junit4] 2> 440756 T2204 C456 P47359 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:35890/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:44717/collection1/&wt=javabin&version=2} {add=[3 (1439090639602974720)]} 0 1
[junit4:junit4] 2> 440756 T2153 C457 P35890 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:44717/collection1/&test.distrib.skip.servers=http://127.0.0.1:49136/collection1/&wt=javabin&version=2} {add=[3 (1439090639602974720)]} 0 5
[junit4:junit4] 2> 440758 T2134 C458 P39279 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 440762 T2134 C458 P39279 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@568ab368 lockFactory=org.apache.lucene.store.NativeFSLockFactory@cd828e0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@568ab368 lockFactory=org.apache.lucene.store.NativeFSLockFactory@cd828e0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 440762 T2134 C458 P39279 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 440764 T2134 C458 P39279 oass.SolrIndexSearcher.<init> Opening Searcher@24cc1fd9 main
[junit4:junit4] 2> 440764 T2134 C458 P39279 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 440765 T2144 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@24cc1fd9 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 440765 T2134 C458 P39279 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 2> 440768 T2154 C457 P35890 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:35890/collection1/, StdNode: http://127.0.0.1:44717/collection1/, StdNode: http://127.0.0.1:49136/collection1/, StdNode: http://127.0.0.1:47359/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 440769 T2156 C457 P35890 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 440770 T2171 C459 P44717 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 440770 T2205 C456 P47359 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 440770 T2187 C460 P49136 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 440774 T2171 C459 P44717 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4ac221b4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d5c615b),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4ac221b4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d5c615b),segFN=segments_2,generation=2}
[junit4:junit4] 2> 440775 T2187 C460 P49136 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@75e0b7cf lockFactory=org.apache.lucene.store.NativeFSLockFactory@450592c7),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@75e0b7cf lockFactory=org.apache.lucene.store.NativeFSLockFactory@450592c7),segFN=segments_2,generation=2}
[junit4:junit4] 2> 440775 T2156 C457 P35890 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6aea1def lockFactory=org.apache.lucene.store.NativeFSLockFactory@4331f521),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6aea1def lockFactory=org.apache.lucene.store.NativeFSLockFactory@4331f521),segFN=segments_2,generation=2}
[junit4:junit4] 2> 440775 T2187 C460 P49136 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 440775 T2171 C459 P44717 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 440776 T2205 C456 P47359 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1cf0d8b4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@584aca85),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1cf0d8b4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@584aca85),segFN=segments_2,generation=2}
[junit4:junit4] 2> 440776 T2156 C457 P35890 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 440777 T2205 C456 P47359 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 440777 T2187 C460 P49136 oass.SolrIndexSearcher.<init> Opening Searcher@5eaa5bb4 main
[junit4:junit4] 2> 440777 T2171 C459 P44717 oass.SolrIndexSearcher.<init> Opening Searcher@51965c17 main
[junit4:junit4] 2> 440778 T2187 C460 P49136 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 440778 T2171 C459 P44717 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 440778 T2195 oasc.JmxMonitoredMap.put WARN Failed to register info bean: searcher javax.management.InstanceAlreadyExistsException: solr/collection1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4] 2> at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:305)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1844)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1721)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2>
[junit4:junit4] 2> 440779 T2156 C457 P35890 oass.SolrIndexSearcher.<init> Opening Searcher@789a12d0 main
[junit4:junit4] 2> 440779 T2178 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@51965c17 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c2)}
[junit4:junit4] 2> 440779 T2205 C456 P47359 oass.SolrIndexSearcher.<init> Opening Searcher@7b563e04 main
[junit4:junit4] 2> 440780 T2195 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5eaa5bb4 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c3)}
[junit4:junit4] 2> 440780 T2156 C457 P35890 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 440781 T2205 C456 P47359 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 440781 T2171 C459 P44717 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4] 2> 440781 T2162 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@789a12d0 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 440781 T2214 oasc.JmxMonitoredMap.put WARN Failed to register info bean: fieldValueCache javax.management.InstanceNotFoundException: solr/collection1:type=fieldValueCache,id=org.apache.solr.search.FastLRUCache
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:138)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4] 2> at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:309)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1844)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1721)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2>
[junit4:junit4] 2> 440782 T2156 C457 P35890 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 13
[junit4:junit4] 2> 440781 T2187 C460 P49136 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4] 2> 440782 T2214 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7b563e04 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 440783 T2205 C456 P47359 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 13
[junit4:junit4] 2> 440783 T2154 C457 P35890 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 16
[junit4:junit4] 2> 440784 T2118 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):330
[junit4:junit4] 2> 440785 T2118 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 440785 T2118 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 440787 T2155 C457 P35890 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 440789 T2170 C459 P44717 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0
[junit4:junit4] 2> 440791 T2188 C460 P49136 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=3 status=0 QTime=0
[junit4:junit4] 2> 440793 T2206 C456 P47359 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 440795 T2118 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 440799 T2186 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4] 2> 440799 T2186 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 440802 T2153 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4] 2> 440803 T2153 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35890/collection1/
[junit4:junit4] 2> 440803 T2153 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:35890 START replicas=[http://127.0.0.1:44717/collection1/, http://127.0.0.1:49136/collection1/, http://127.0.0.1:47359/collection1/] nUpdates=100
[junit4:junit4] 2> 440805 T2169 C459 P44717 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 440805 T2203 C456 P47359 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 440805 T2153 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:35890 Received 3 versions from 127.0.0.1:44717/collection1/
[junit4:junit4] 2> 440805 T2187 C460 P49136 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 440806 T2153 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:35890 Our versions are newer. ourLowThreshold=1439090639551594496 otherHigh=1439090639577808896
[junit4:junit4] 2> 440806 T2153 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:35890 Received 5 versions from 127.0.0.1:47359/collection1/
[junit4:junit4] 2> 440806 T2153 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:35890 Our versions are newer. ourLowThreshold=1439090639551594496 otherHigh=1439090639592488960
[junit4:junit4] 2> 440807 T2153 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:35890 Received 4 versions from 127.0.0.1:49136/collection1/
[junit4:junit4] 2> 440807 T2153 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:35890 Our versions are newer. ourLowThreshold=1439090639551594496 otherHigh=1439090639592488960
[junit4:junit4] 2> 440807 T2153 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:35890 DONE. sync succeeded
[junit4:junit4] 2> 440807 T2153 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 440807 T2153 oasc.SyncStrategy.syncToMe http://127.0.0.1:35890/collection1/: try and ask http://127.0.0.1:44717/collection1/ to sync
[junit4:junit4] 2> 440808 T2153 oasc.SyncStrategy.syncToMe http://127.0.0.1:35890/collection1/: try and ask http://127.0.0.1:49136/collection1/ to sync
[junit4:junit4] 2> 440808 T2153 oasc.SyncStrategy.syncToMe http://127.0.0.1:35890/collection1/: try and ask http://127.0.0.1:47359/collection1/ to sync
[junit4:junit4] 2> 440809 T2170 C459 P44717 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44717 START replicas=[http://127.0.0.1:35890/collection1/] nUpdates=100
[junit4:junit4] 2> 440809 T2188 C460 P49136 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:49136 START replicas=[http://127.0.0.1:35890/collection1/] nUpdates=100
[junit4:junit4] 2> 440809 T2204 C456 P47359 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:47359 START replicas=[http://127.0.0.1:35890/collection1/] nUpdates=100
[junit4:junit4] 2> 440810 T2156 C457 P35890 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 440811 T2154 C457 P35890 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 440811 T2188 C460 P49136 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:49136 Received 5 versions from 127.0.0.1:35890/collection1/
[junit4:junit4] 2> 440811 T2204 C456 P47359 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:47359 Received 5 versions from 127.0.0.1:35890/collection1/
[junit4:junit4] 2> 440811 T2188 C460 P49136 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:49136 Requesting updates from 127.0.0.1:35890/collection1/n=1 versions=[1439090639602974720]
[junit4:junit4] 2> 440812 T2155 C457 P35890 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 440812 T2204 C456 P47359 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:47359 Our versions are newer. ourLowThreshold=1439090639551594496 otherHigh=1439090639592488960
[junit4:junit4] 2> 440813 T2204 C456 P47359 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:47359 DONE. sync succeeded
[junit4:junit4] 2> 440812 T2170 C459 P44717 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44717 Received 5 versions from 127.0.0.1:35890/collection1/
[junit4:junit4] 2> 440813 T2204 C456 P47359 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:35890/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=4
[junit4:junit4] 2> 440813 T2170 C459 P44717 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:44717 Requesting updates from 127.0.0.1:35890/collection1/n=2 versions=[1439090639602974720, 1439090639592488960]
[junit4:junit4] 2> 440814 T2153 oasc.SyncStrategy.syncToMe http://127.0.0.1:35890/collection1/: sync completed with http://127.0.0.1:47359/collection1/
[junit4:junit4] 2> 440814 T2156 C457 P35890 oasc.SolrCore.execute [collection1] webapp= path=/get params={distrib=false&getUpdates=1439090639602974720&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 440816 T2154 C457 P35890 oasc.SolrCore.execute [collection1] webapp= path=/get params={distrib=false&getUpdates=1439090639602974720,1439090639592488960&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 440816 T2188 C460 P49136 oasup.LogUpdateProcessor.finish [collection1] {add=[3 (1439090639602974720)]} 0 1
[junit4:junit4] 2> 440817 T2188 C460 P49136 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:49136 DONE. sync succeeded
[junit4:junit4] 2> 440817 T2188 C460 P49136 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:35890/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=8
[junit4:junit4] 2> 440818 T2153 oasc.SyncStrategy.syncToMe http://127.0.0.1:35890/collection1/: sync completed with http://127.0.0.1:49136/collection1/
[junit4:junit4] 2> 440819 T2170 C459 P44717 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1439090639592488960), 3 (1439090639602974720)]} 0 3
[junit4:junit4] 2> 440820 T2170 C459 P44717 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44717 DONE. sync succeeded
[junit4:junit4] 2> 440820 T2170 C459 P44717 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:35890/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=11
[junit4:junit4] 2> 440821 T2153 oasc.SyncStrategy.syncToMe http://127.0.0.1:35890/collection1/: sync completed with http://127.0.0.1:44717/collection1/
[junit4:junit4] 2> 440821 T2153 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=19
[junit4:junit4] 2> 440821 T2186 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=23
[junit4:junit4] 2> 440822 T2118 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 440822 T2118 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 440823 T2118 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 440823 T2118 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 440825 T2133 C458 P39279 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 440826 T2133 C458 P39279 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 440826 T2133 C458 P39279 oass.SolrIndexSearcher.<init> Opening Searcher@2456def8 main
[junit4:junit4] 2> 440827 T2133 C458 P39279 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 440828 T2144 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2456def8 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 440828 T2133 C458 P39279 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4] 2> 440830 T2155 C457 P35890 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:35890/collection1/, StdNode: http://127.0.0.1:44717/collection1/, StdNode: http://127.0.0.1:49136/collection1/, StdNode: http://127.0.0.1:47359/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 440832 T2189 C460 P49136 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 440832 T2172 C459 P44717 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 440832 T2156 C457 P35890 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 440834 T2156 C457 P35890 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 440834 T2205 C456 P47359 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 440834 T2156 C457 P35890 oass.SolrIndexSearcher.<init> Opening Searcher@15d337cb main
[junit4:junit4] 2> 440834 T2205 C456 P47359 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 440835 T2156 C457 P35890 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 440835 T2205 C456 P47359 oass.SolrIndexSearcher.<init> Opening Searcher@340a3fa8 main
[junit4:junit4] 2> 440836 T2205 C456 P47359 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 440836 T2162 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@15d337cb main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 440837 T2189 C460 P49136 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@75e0b7cf lockFactory=org.apache.lucene.store.NativeFSLockFactory@450592c7),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@75e0b7cf lockFactory=org.apache.lucene.store.NativeFSLockFactory@450592c7),segFN=segments_3,generation=3}
[junit4:junit4] 2> 440838 T2189 C460 P49136 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 440838 T2214 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@340a3fa8 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 440838 T2156 C457 P35890 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 440839 T2172 C459 P44717 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4ac221b4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d5c615b),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4ac221b4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d5c615b),segFN=segments_3,generation=3}
[junit4:junit4] 2> 440839 T2189 C460 P49136 oass.SolrIndexSearcher.<init> Opening Searcher@b81a33b main
[junit4:junit4] 2> 440839 T2205 C456 P47359 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 440839 T2172 C459 P44717 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 440840 T2189 C460 P49136 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 440841 T2195 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b81a33b main{StandardDirectoryReader(segments_3:5:nrt _0(5.0):c3 _1(5.0):c1)}
[junit4:junit4] 2> 440841 T2189 C460 P49136 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4] 2> 440842 T2172 C459 P44717 oass.SolrIndexSearcher.<init> Opening Searcher@6b062cc7 main
[junit4:junit4] 2> 440842 T2172 C459 P44717 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 440843 T2178 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6b062cc7 main{StandardDirectoryReader(segments_3:5:nrt _0(5.0):c2 _1(5.0):c2)}
[junit4:junit4] 2> 440844 T2172 C459 P44717 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 12
[junit4:junit4] 2> 440844 T2155 C457 P35890 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 14
[junit4:junit4] 2> 440845 T2118 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 440848 T2154 C457 P35890 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 440850 T2169 C459 P44717 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 440852 T2187 C460 P49136 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 440854 T2206 C456 P47359 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 442855 T2118 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> check const of shard1
[junit4:junit4] 2> client0
[junit4:junit4] 2> PROPS:core_node1:{"state":"active","core":"collection1","node_name":"127.0.0.1:35890_","base_url":"http://127.0.0.1:35890","leader":"true"}
[junit4:junit4] 2> 442858 T2155 C457 P35890 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client1
[junit4:junit4] 2> PROPS:core_node2:{"state":"active","core":"collection1","node_name":"127.0.0.1:44717_","base_url":"http://127.0.0.1:44717"}
[junit4:junit4] 2> 442860 T2169 C459 P44717 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client2
[junit4:junit4] 2> PROPS:core_node3:{"state":"active","core":"collection1","node_name":"127.0.0.1:49136_","base_url":"http://127.0.0.1:49136"}
[junit4:junit4] 2> 442862 T2189 C460 P49136 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client3
[junit4:junit4] 2> PROPS:core_node4:{"state":"active","core":"collection1","node_name":"127.0.0.1:47359_","base_url":"http://127.0.0.1:47359"}
[junit4:junit4] 2> 442864 T2205 C456 P47359 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> 442866 T2153 C457 P35890 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&wt=javabin&version=2} hits=4 status=0 QTime=0
[junit4:junit4] 2> 442870 T2133 C458 P39279 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1439090641821761536)]} 0 2
[junit4:junit4] 2> 442875 T2156 C457 P35890 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:47359/collection1/]
[junit4:junit4] 2> 442875 T2156 C457 P35890 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44717/collection1/ against:[http://127.0.0.1:47359/collection1/] result:false
[junit4:junit4] 2> 442875 T2156 C457 P35890 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:49136/collection1/ against:[http://127.0.0.1:47359/collection1/] result:false
[junit4:junit4] 2> 442875 T2156 C457 P35890 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:47359/collection1/ against:[http://127.0.0.1:47359/collection1/] result:true
[junit4:junit4] 2> 442880 T2186 C460 P49136 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:35890/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:47359/collection1/&wt=javabin&version=2} {add=[4 (1439090641829101568)]} 0 2
[junit4:junit4] 2> 442880 T2170 C459 P44717 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:35890/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:47359/collection1/&wt=javabin&version=2} {add=[4 (1439090641829101568)]} 0 1
[junit4:junit4] 2> 442881 T2156 C457 P35890 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:49136/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:47359/collection1/&wt=javabin&version=2} {add=[4 (1439090641829101568)]} 0 7
[junit4:junit4] 2> 442881 T2188 C460 P49136 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:47359/collection1/&wt=javabin&version=2} {add=[4]} 0 8
[junit4:junit4] 2> 442882 T2118 oasc.ChaosMonkey.monkeyLog monkey: kill shard! 35890
[junit4:junit4] 2> 442903 T2118 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1465730992
[junit4:junit4] 2> 443744 T2141 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 443745 T2141 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:35890_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35890"}
[junit4:junit4] 2> 443745 T2141 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4:junit4] 2> 443749 T2193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 443749 T2140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 443749 T2176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 443749 T2160 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 443749 T2210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 443749 T2147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 444906 T2118 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 444906 T2118 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 444909 T2118 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@17ac4e1a
[junit4:junit4] 2> 444912 T2118 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=5,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 444913 T2118 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 444914 T2118 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 444914 T2118 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> C457_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:35890_, base_url=http://127.0.0.1:35890, leader=true}
[junit4:junit4] 2> 444917 T2118 C457 P35890 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6aea1def lockFactory=org.apache.lucene.store.NativeFSLockFactory@4331f521),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6aea1def lockFactory=org.apache.lucene.store.NativeFSLockFactory@4331f521),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6aea1def lockFactory=org.apache.lucene.store.NativeFSLockFactory@4331f521),segFN=segments_3,generation=3}
[junit4:junit4] 2> 444918 T2118 C457 P35890 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 444920 T2118 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 444920 T2118 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 444920 T2118 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty1 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty1;done=false>>]
[junit4:junit4] 2> 444921 T2118 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty1
[junit4:junit4] 2> 444921 T2118 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty1/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty1/index;done=false>>]
[junit4:junit4] 2> 444921 T2118 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1372423760296/jetty1/index
[junit4:junit4] 2> 444923 T2140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 444923 T2210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 444924 T2147 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 444924 T2193 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 444924 T2118 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 444924 T2147 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 444925 T2210 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 444924 T2193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 444925 T2140 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 444925 T2176 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4] 2> 444927 T2176 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 444927 T2176 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 444927 T2176 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 444927 T2176 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44717/collection1/
[junit4:junit4] 2> 444927 T2176 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44717 START replicas=[http://127.0.0.1:35890/collection1/, http://127.0.0.1:49136/collection1/, http://127.0.0.1:47359/collection1/] nUpdates=100
[junit4:junit4] 2> 444929 T2176 oasu.PeerSync.handleResponse WARN PeerSync: core=collection1 url=http://127.0.0.1:44717 couldn't connect to http://127.0.0.1:35890/collection1/, counting as success
[junit4:junit4] 2> 444930 T2205 C456 P47359 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 444930 T2187 C460 P49136 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 444930 T2176 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44717 Received 5 versions from 127.0.0.1:47359/collection1/
[junit4:junit4] 2> 444931 T2176 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44717 Our versions are newer. ourLowThreshold=1439090639564177408 otherHigh=1439090639592488960
[junit4:junit4] 2> 444931 T2176 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44717 Received 6 versions from 127.0.0.1:49136/collection1/
[junit4:junit4] 2> 444931 T2176 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44717 Our versions are newer. ourLowThreshold=1439090639564177408 otherHigh=1439090639602974720
[junit4:junit4] 2> 444931 T2176 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44717 DONE. sync succeeded
[junit4:junit4] 2> 444931 T2176 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 444931 T2176 oasc.SyncStrategy.syncToMe http://127.0.0.1:44717/collection1/: try and ask http://127.0.0.1:35890/collection1/ to sync
[junit4:junit4] 2> 444932 T2176 oasc.SyncStrategy.syncToMe http://127.0.0.1:44717/collection1/: try and ask http://127.0.0.1:49136/collection1/ to sync
[junit4:junit4] 2> 444932 T2176 oasc.SyncStrategy.syncToMe http://127.0.0.1:44717/collection1/: try and ask http://127.0.0.1:47359/collection1/ to sync
[junit4:junit4] 2> 444932 T2176 oasc.SolrException.log ERROR Sync request error: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:35890/collection1
[junit4:junit4] 2> 444932 T2176 oasc.SyncStrategy.syncToMe http://127.0.0.1:44717/collection1/: Sync failed - asking replica (http://127.0.0.1:35890/collection1/) to recover.
[junit4:junit4] 2> 444933 T2243 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 444934 T2189 C460 P49136 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:49136 START replicas=[http://127.0.0.1:44717/collection1/] nUpdates=100
[junit4:junit4] 2> 444934 T2204 C456 P47359 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:47359 START replicas=[http://127.0.0.1:44717/collection1/] nUpdates=100
[junit4:junit4] 2> 444935 T2172 C459 P44717 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 444936 T2204 C456 P47359 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:47359 Received 6 versions from 127.0.0.1:44717/collection1/
[junit4:junit4] 2> 444936 T2204 C456 P47359 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:47359 Requesting updates from 127.0.0.1:44717/collection1/n=1 versions=[1439090641829101568]
[junit4:junit4] 2> 444936 T2171 C459 P44717 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 444936 T2243 oasc.SolrException.log ERROR http://127.0.0.1:44717/collection1/: Could not tell a replica to recover:org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:35890
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4] 2> at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:297)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:35890 refused
[junit4:junit4] 2> at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
[junit4:junit4] 2> at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4:junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4:junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4:junit4] 2> ... 5 more
[junit4:junit4] 2> Caused by: java.net.ConnectException: Connection refused
[junit4:junit4] 2> at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4:junit4] 2> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4:junit4] 2> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4:junit4] 2> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4:junit4] 2> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[junit4:junit4] 2> at java.net.Socket.connect(Socket.java:579)
[junit4:junit4] 2> at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4:junit4] 2> at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4:junit4] 2> ... 12 more
[junit4:junit4] 2>
[junit4:junit4] 2> 444937 T2189 C460 P49136 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:49136 Received 6 versions from 127.0.0.1:44717/collection1/
[junit4:junit4] 2> 444937 T2189 C460 P49136 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:49136 Our versions are newer. ourLowThreshold=1439090639564177408 otherHigh=1439090639602974720
[junit4:junit4] 2> 444937 T2189 C460 P49136 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:49136 DONE. sync succeeded
[junit4:junit4] 2> 444938 T2189 C460 P49136 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:44717/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=5
[junit4:junit4] 2> 444938 T2170 C459 P44717 oasc.SolrCore.exe
[...truncated too long message...]
"node_name":"127.0.0.1:49136_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:49136"},
[junit4:junit4] 1> "core_node4":{
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:47359_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:47359",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"core_node1":{
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:39279_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:39279",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:49136_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:47359_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:35890_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:39279_ (0)
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/collections (2)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89943163554889738-core_node3-n_0000000005 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89943163554889742-core_node1-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89943163554889740-core_node4-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (1)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:47359_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:47359"}
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89943163554889731-core_node1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:39279_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:39279"}
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/election (4)
[junit4:junit4] 1> /solr/overseer_elect/election/89943163554889742-127.0.0.1:35890_-n_0000000005 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89943163554889740-127.0.0.1:47359_-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89943163554889731-127.0.0.1:39279_-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89943163554889738-127.0.0.1:49136_-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89943163554889731-127.0.0.1:39279_-n_0000000000"}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=BD3613953711EFA1 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=lt_LT -Dtests.timezone=Asia/Pontianak -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE 98.2s J1 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:35890/collection1lastClient and got 249 from http://127.0.0.1:49136/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([BD3613953711EFA1:3CD09D8D404E8F9D]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1045)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:237)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 517290 T2118 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 98221 T2117 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
[junit4:junit4] 2> 517410 T2193 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 517410 T2193 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 517410 T2193 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Asserting, sim=DefaultSimilarity, locale=lt_LT, timezone=Asia/Pontianak
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=8,threads=1,free=292983552,total=468713472
[junit4:junit4] 2> NOTE: All tests run in this JVM: [FileUtilsTest, XmlUpdateRequestHandlerTest, IndexSchemaRuntimeFieldTest, DateMathParserTest, StatsComponentTest, TestCSVLoader, IndexBasedSpellCheckerTest, HdfsSyncSliceTest, DistanceFunctionTest, EchoParamsTest, DebugComponentTest, TestCollationField, SolrRequestParserTest, TestQuerySenderNoQuery, TestIndexSearcher, CurrencyFieldXmlFileTest, SimplePostToolTest, HdfsUnloadDistributedZkTest, TestDistributedSearch, TestComponentsName, TestSchemaSimilarityResource, CopyFieldTest, TestSchemaResource, TestFastOutputStream, NumericFieldsTest, HdfsRecoveryZkTest, TestDefaultSimilarityFactory, OutputWriterTest, TestManagedSchema, TestUpdate, TimeZoneUtilsTest, StandardRequestHandlerTest, TestArbitraryIndexDir, TestSuggestSpellingConverter, TermVectorComponentDistributedTest, TestBinaryField, SpellCheckComponentTest, HdfsCollectionsAPIDistributedZkTest, TestReloadAndDeleteDocs, ChaosMonkeySafeLeaderTest, DocValuesMultiTest, TestDFRSimilarityFactory, SolrPluginUtilsTest, TestCloudManagedSchema, PingRequestHandlerTest, OpenExchangeRatesOrgProviderTest, MultiTermTest, TestReload, TestCloudManagedSchemaAddField, HdfsBasicDistributedZk2Test, TestQueryTypes, DOMUtilTest, TestCSVResponseWriter, JSONWriterTest, TestNumberUtils, SpellPossibilityIteratorTest, TestPartialUpdateDeduplication, SolrCoreCheckLockOnStartupTest, ShardRoutingCustomTest, DirectSolrSpellCheckerTest, TestUtils, DocumentAnalysisRequestHandlerTest, TestRangeQuery, UpdateParamsTest, TestJmxIntegration, TestFoldingMultitermQuery, TestUniqueKeyFieldResource, TestFieldCollectionResource, URLClassifyProcessorTest, TestFiltering, TestSolrJ, IndexReaderFactoryTest, BasicDistributedZk2Test, SyncSliceTest]
[junit4:junit4] Completed on J1 in 98.74s, 1 test, 1 failure <<< FAILURES!
[...truncated 488 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:190: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:443: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1245: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:889: There were test failures: 312 suites, 1316 tests, 1 failure, 20 ignored (13 assumptions)
Total time: 41 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_25 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_25) - Build #
6349 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6349/
Java: 64bit/jdk1.7.0_25 -XX:+UseCompressedOops -XX:+UseG1GC
1 tests failed.
REGRESSION: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:45675/uw_b/cf/collection1lastClient and got 261 from http://127.0.0.1:44231/uw_b/cf/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:45675/uw_b/cf/collection1lastClient and got 261 from http://127.0.0.1:44231/uw_b/cf/collection1
at __randomizedtesting.SeedInfo.seed([4C71556C638378C7:CD97DB7414DC18FB]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1045)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:237)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9268 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4] 2> 57167 T231 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /uw_b/cf
[junit4:junit4] 2> 57175 T231 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1372418129605
[junit4:junit4] 2> 57176 T231 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 57177 T232 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 57277 T231 oasc.ZkTestServer.run start zk server on port:53479
[junit4:junit4] 2> 57278 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 57283 T238 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45ea5f99 name:ZooKeeperConnection Watcher:127.0.0.1:53479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 57283 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 57284 T231 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 57288 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 57289 T240 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d476c50 name:ZooKeeperConnection Watcher:127.0.0.1:53479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 57290 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 57290 T231 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 57294 T231 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 57299 T231 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 57303 T231 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 57306 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 57307 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 57312 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 57313 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 57317 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 57318 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 57321 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 57322 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 57325 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 57326 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 57330 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 57330 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 57334 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 57334 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 57338 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 57339 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 57342 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 57343 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 57434 T231 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 57438 T231 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49202
[junit4:junit4] 2> 57438 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 57439 T231 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 57439 T231 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799
[junit4:junit4] 2> 57439 T231 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799/solr.xml
[junit4:junit4] 2> 57440 T231 oasc.CoreContainer.<init> New CoreContainer 1792687692
[junit4:junit4] 2> 57440 T231 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799/'
[junit4:junit4] 2> 57440 T231 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799/'
[junit4:junit4] 2> 57485 T231 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 57486 T231 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 57486 T231 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 57487 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 57487 T231 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 57487 T231 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 57488 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 57488 T231 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 57488 T231 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 57489 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 57493 T231 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 57494 T231 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53479/solr
[junit4:junit4] 2> 57494 T231 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 57495 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 57497 T251 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f0e8369 name:ZooKeeperConnection Watcher:127.0.0.1:53479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 57497 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 57499 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 57506 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 57509 T253 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6efe55ac name:ZooKeeperConnection Watcher:127.0.0.1:53479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 57509 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 57512 T231 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 57517 T231 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 57521 T231 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 57523 T231 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49202_uw_b%2Fcf
[junit4:junit4] 2> 57525 T231 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49202_uw_b%2Fcf
[junit4:junit4] 2> 57529 T231 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 57535 T231 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 57540 T231 oasc.Overseer.start Overseer (id=89942794542055427-127.0.0.1:49202_uw_b%2Fcf-n_0000000000) starting
[junit4:junit4] 2> 57544 T231 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 57549 T255 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 57549 T231 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 57552 T231 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 57554 T231 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 57557 T254 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 57560 T256 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 57560 T256 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 57561 T256 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 59063 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 59064 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49202_uw_b%2Fcf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49202/uw_b/cf"}
[junit4:junit4] 2> 59064 T254 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 59065 T254 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 59071 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 59562 T256 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 59562 T256 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799/collection1
[junit4:junit4] 2> 59562 T256 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 59564 T256 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 59564 T256 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 59566 T256 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799/collection1/'
[junit4:junit4] 2> 59566 T256 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799/collection1/lib/README' to classloader
[junit4:junit4] 2> 59567 T256 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 59594 T256 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 59639 T256 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 59641 T256 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 59650 T256 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 60032 T256 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 60053 T256 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 60058 T256 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 60081 T256 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 60085 T256 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 60089 T256 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 60091 T256 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 60091 T256 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 60092 T256 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 60093 T256 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 60093 T256 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 60094 T256 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 60094 T256 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372418129605/control/data/
[junit4:junit4] 2> 60094 T256 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@150858bb
[junit4:junit4] 2> 60095 T256 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/control/data
[junit4:junit4] 2> 60095 T256 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372418129605/control/data/index/
[junit4:junit4] 2> 60095 T256 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372418129605/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 60096 T256 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/control/data/index
[junit4:junit4] 2> 60097 T256 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@211782bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@8504643),segFN=segments_1,generation=1}
[junit4:junit4] 2> 60098 T256 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 60100 T256 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 60100 T256 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 60101 T256 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 60101 T256 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 60102 T256 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 60102 T256 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 60102 T256 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 60103 T256 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 60103 T256 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 60105 T256 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 60108 T256 oass.SolrIndexSearcher.<init> Opening Searcher@10279ee2 main
[junit4:junit4] 2> 60109 T256 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 60109 T256 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 60113 T257 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@10279ee2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 60115 T256 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 60116 T256 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49202/uw_b/cf collection:control_collection shard:shard1
[junit4:junit4] 2> 60117 T256 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 60126 T256 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4] 2> 60129 T256 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 60129 T256 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 60129 T256 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49202/uw_b/cf/collection1/
[junit4:junit4] 2> 60130 T256 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 60130 T256 oasc.SyncStrategy.syncToMe http://127.0.0.1:49202/uw_b/cf/collection1/ has no replicas
[junit4:junit4] 2> 60130 T256 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49202/uw_b/cf/collection1/ shard1
[junit4:junit4] 2> 60130 T256 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 60577 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 60587 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 60638 T256 oasc.ZkController.register We are http://127.0.0.1:49202/uw_b/cf/collection1/ and leader is http://127.0.0.1:49202/uw_b/cf/collection1/
[junit4:junit4] 2> 60638 T256 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49202/uw_b/cf
[junit4:junit4] 2> 60638 T256 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 60639 T256 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 60639 T256 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 60641 T256 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 60642 T231 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 60643 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 60643 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 60647 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 60650 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 60651 T260 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@785d564b name:ZooKeeperConnection Watcher:127.0.0.1:53479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 60652 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 60653 T231 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 60657 T231 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 60721 T231 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 60723 T231 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45675
[junit4:junit4] 2> 60724 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 60725 T231 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 60725 T231 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087
[junit4:junit4] 2> 60725 T231 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087/solr.xml
[junit4:junit4] 2> 60726 T231 oasc.CoreContainer.<init> New CoreContainer 2095146885
[junit4:junit4] 2> 60726 T231 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087/'
[junit4:junit4] 2> 60726 T231 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087/'
[junit4:junit4] 2> 60770 T231 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 60771 T231 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 60771 T231 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 60772 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 60772 T231 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 60772 T231 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 60773 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 60773 T231 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 60773 T231 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 60774 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 60777 T231 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 60778 T231 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53479/solr
[junit4:junit4] 2> 60778 T231 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 60779 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 60782 T271 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@73918082 name:ZooKeeperConnection Watcher:127.0.0.1:53479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 60782 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 60784 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 60789 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 60790 T273 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b480442 name:ZooKeeperConnection Watcher:127.0.0.1:53479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 60791 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 60796 T231 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 61801 T231 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45675_uw_b%2Fcf
[junit4:junit4] 2> 61802 T231 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45675_uw_b%2Fcf
[junit4:junit4] 2> 61806 T260 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 61807 T253 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 61807 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 61807 T273 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 61814 T274 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 61815 T274 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 61817 T274 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 62094 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 62095 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49202_uw_b%2Fcf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49202/uw_b/cf"}
[junit4:junit4] 2> 62100 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45675_uw_b%2Fcf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45675/uw_b/cf"}
[junit4:junit4] 2> 62100 T254 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 62100 T254 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 62105 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 62106 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 62106 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 62818 T274 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 62818 T274 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087/collection1
[junit4:junit4] 2> 62818 T274 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 62819 T274 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 62820 T274 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 62821 T274 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087/collection1/'
[junit4:junit4] 2> 62822 T274 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087/collection1/lib/README' to classloader
[junit4:junit4] 2> 62822 T274 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 62849 T274 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 62910 T274 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 62916 T274 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 62924 T274 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 63311 T274 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 63317 T274 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 63319 T274 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 63348 T274 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 63353 T274 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 63359 T274 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 63361 T274 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 63361 T274 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 63362 T274 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 63363 T274 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 63364 T274 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 63364 T274 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 63365 T274 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1/
[junit4:junit4] 2> 63365 T274 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@150858bb
[junit4:junit4] 2> 63366 T274 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1
[junit4:junit4] 2> 63366 T274 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1/index/
[junit4:junit4] 2> 63367 T274 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 63367 T274 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1/index
[junit4:junit4] 2> 63370 T274 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2179600a lockFactory=org.apache.lucene.store.NativeFSLockFactory@6cda313c),segFN=segments_1,generation=1}
[junit4:junit4] 2> 63370 T274 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 63374 T274 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 63374 T274 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 63375 T274 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 63376 T274 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 63377 T274 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 63377 T274 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 63378 T274 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 63379 T274 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 63379 T274 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 63383 T274 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 63387 T274 oass.SolrIndexSearcher.<init> Opening Searcher@223e4532 main
[junit4:junit4] 2> 63388 T274 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 63388 T274 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 63395 T275 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@223e4532 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 63398 T274 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 63399 T274 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45675/uw_b/cf collection:collection1 shard:shard1
[junit4:junit4] 2> 63400 T274 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 63410 T274 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4] 2> 63413 T274 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 63414 T274 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 63414 T274 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4] 2> 63415 T274 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 63415 T274 oasc.SyncStrategy.syncToMe http://127.0.0.1:45675/uw_b/cf/collection1/ has no replicas
[junit4:junit4] 2> 63415 T274 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45675/uw_b/cf/collection1/ shard1
[junit4:junit4] 2> 63416 T274 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 63614 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 63623 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 63623 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 63623 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 63675 T274 oasc.ZkController.register We are http://127.0.0.1:45675/uw_b/cf/collection1/ and leader is http://127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4] 2> 63675 T274 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45675/uw_b/cf
[junit4:junit4] 2> 63676 T274 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 63676 T274 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 63676 T274 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 63679 T274 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 63681 T231 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 63681 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 63682 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 63749 T231 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 63751 T231 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44900
[junit4:junit4] 2> 63752 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 63752 T231 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 63753 T231 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115
[junit4:junit4] 2> 63753 T231 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115/solr.xml
[junit4:junit4] 2> 63753 T231 oasc.CoreContainer.<init> New CoreContainer 226056211
[junit4:junit4] 2> 63754 T231 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115/'
[junit4:junit4] 2> 63754 T231 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115/'
[junit4:junit4] 2> 63808 T231 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 63809 T231 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 63809 T231 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 63810 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 63810 T231 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 63810 T231 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 63811 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 63811 T231 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 63811 T231 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 63812 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 63815 T231 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 63816 T231 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53479/solr
[junit4:junit4] 2> 63816 T231 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 63817 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 63819 T287 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76c86b31 name:ZooKeeperConnection Watcher:127.0.0.1:53479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 63819 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 63821 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 63825 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 63826 T289 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50021d3 name:ZooKeeperConnection Watcher:127.0.0.1:53479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 63826 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 63831 T231 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 64834 T231 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44900_uw_b%2Fcf
[junit4:junit4] 2> 64836 T231 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44900_uw_b%2Fcf
[junit4:junit4] 2> 64839 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 64839 T253 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 64840 T260 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 64840 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 64840 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 64840 T289 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 64842 T273 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 64846 T290 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 64846 T290 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 64848 T290 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 65131 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 65132 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45675_uw_b%2Fcf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45675/uw_b/cf"}
[junit4:junit4] 2> 65136 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44900_uw_b%2Fcf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44900/uw_b/cf"}
[junit4:junit4] 2> 65137 T254 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 65137 T254 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 65142 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 65142 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 65142 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 65142 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 65848 T290 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 65849 T290 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115/collection1
[junit4:junit4] 2> 65849 T290 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 65850 T290 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 65850 T290 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 65852 T290 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115/collection1/'
[junit4:junit4] 2> 65852 T290 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115/collection1/lib/README' to classloader
[junit4:junit4] 2> 65853 T290 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 65879 T290 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 65909 T290 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 65910 T290 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 65916 T290 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 66350 T290 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 66358 T290 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 66360 T290 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 66385 T290 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 66389 T290 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 66394 T290 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 66395 T290 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 66396 T290 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 66396 T290 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 66398 T290 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 66398 T290 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 66398 T290 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 66398 T290 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty2/
[junit4:junit4] 2> 66399 T290 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@150858bb
[junit4:junit4] 2> 66400 T290 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty2
[junit4:junit4] 2> 66400 T290 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty2/index/
[junit4:junit4] 2> 66400 T290 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 66401 T290 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty2/index
[junit4:junit4] 2> 66402 T290 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@64f56312 lockFactory=org.apache.lucene.store.NativeFSLockFactory@50b7fbb8),segFN=segments_1,generation=1}
[junit4:junit4] 2> 66402 T290 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 66405 T290 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 66405 T290 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 66406 T290 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 66407 T290 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 66408 T290 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 66409 T290 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 66409 T290 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 66410 T290 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 66411 T290 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 66415 T290 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 66418 T290 oass.SolrIndexSearcher.<init> Opening Searcher@24803429 main
[junit4:junit4] 2> 66419 T290 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 66420 T290 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 66428 T291 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@24803429 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 66431 T290 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 66432 T290 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44900/uw_b/cf collection:collection1 shard:shard1
[junit4:junit4] 2> 66435 T290 oasc.ZkController.register We are http://127.0.0.1:44900/uw_b/cf/collection1/ and leader is http://127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4] 2> 66436 T290 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44900/uw_b/cf
[junit4:junit4] 2> 66436 T290 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 66436 T290 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C21 name=collection1 org.apache.solr.core.SolrCore@7c20a9de url=http://127.0.0.1:44900/uw_b/cf/collection1 node=127.0.0.1:44900_uw_b%2Fcf C21_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:44900_uw_b%2Fcf, base_url=http://127.0.0.1:44900/uw_b/cf}
[junit4:junit4] 2> 66440 T292 C21 P44900 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 66441 T290 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 66441 T292 C21 P44900 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 66442 T292 C21 P44900 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 66442 T292 C21 P44900 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 66443 T231 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 66443 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 66444 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 66444 T292 C21 P44900 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 66454 T267 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 66550 T231 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 66553 T231 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44231
[junit4:junit4] 2> 66553 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 66554 T231 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 66554 T231 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879
[junit4:junit4] 2> 66554 T231 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879/solr.xml
[junit4:junit4] 2> 66555 T231 oasc.CoreContainer.<init> New CoreContainer 445244580
[junit4:junit4] 2> 66555 T231 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879/'
[junit4:junit4] 2> 66556 T231 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879/'
[junit4:junit4] 2> 66635 T231 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 66635 T231 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 66636 T231 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 66636 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 66636 T231 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 66636 T231 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 66637 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 66637 T231 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 66637 T231 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 66638 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 66642 T231 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 66642 T231 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53479/solr
[junit4:junit4] 2> 66643 T231 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 66643 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 66645 T304 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b16555b name:ZooKeeperConnection Watcher:127.0.0.1:53479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 66646 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 66647 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 66650 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 66651 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44900_uw_b%2Fcf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44900/uw_b/cf"}
[junit4:junit4] 2> 66652 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 66654 T306 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@632099ee name:ZooKeeperConnection Watcher:127.0.0.1:53479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 66656 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 66657 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 66657 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 66657 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 66657 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 66663 T231 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 67455 T267 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 67456 T267 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node2&state=recovering&nodeName=127.0.0.1:44900_uw_b%252Fcf&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4] 2> 67667 T231 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44231_uw_b%2Fcf
[junit4:junit4] 2> 67670 T231 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44231_uw_b%2Fcf
[junit4:junit4] 2> 67673 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 67674 T260 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 67674 T253 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 67674 T289 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 67674 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 67674 T306 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 67674 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 67674 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 67677 T273 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 67681 T307 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 67681 T307 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 67683 T307 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 68165 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 68166 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44231_uw_b%2Fcf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44231/uw_b/cf"}
[junit4:junit4] 2> 68167 T254 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 68167 T254 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 68172 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 68172 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 68172 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 68172 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 68172 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 68683 T307 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 68684 T307 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879/collection1
[junit4:junit4] 2> 68684 T307 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 68685 T307 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 68685 T307 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 68687 T307 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879/collection1/'
[junit4:junit4] 2> 68688 T307 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879/collection1/lib/README' to classloader
[junit4:junit4] 2> 68688 T307 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 68717 T307 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 68750 T307 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 68752 T307 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 68758 T307 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 69131 T307 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 69137 T307 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 69139 T307 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 69160 T307 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 69167 T307 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 69170 T307 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 69172 T307 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 69173 T307 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 69173 T307 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 69175 T307 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 69175 T307 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 69175 T307 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 69176 T307 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty3/
[junit4:junit4] 2> 69176 T307 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@150858bb
[junit4:junit4] 2> 69177 T307 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty3
[junit4:junit4] 2> 69177 T307 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty3/index/
[junit4:junit4] 2> 69177 T307 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 69177 T307 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty3/index
[junit4:junit4] 2> 69179 T307 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3267b8c3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4389c038),segFN=segments_1,generation=1}
[junit4:junit4] 2> 69179 T307 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 69181 T307 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 69181 T307 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 69182 T307 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 69183 T307 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 69183 T307 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 69183 T307 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 69184 T307 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 69184 T307 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 69185 T307 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 69187 T307 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 69190 T307 oass.SolrIndexSearcher.<init> Opening Searcher@3540be79 main
[junit4:junit4] 2> 69190 T307 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 69190 T307 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 69195 T308 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3540be79 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 69198 T307 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 69198 T307 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44231/uw_b/cf collection:collection1 shard:shard1
[junit4:junit4] 2> 69202 T307 oasc.ZkController.register We are http://127.0.0.1:44231/uw_b/cf/collection1/ and leader is http://127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4] 2> 69202 T307 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44231/uw_b/cf
[junit4:junit4] 2> 69203 T307 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 69203 T307 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C22 name=collection1 org.apache.solr.core.SolrCore@4b2ea730 url=http://127.0.0.1:44231/uw_b/cf/collection1 node=127.0.0.1:44231_uw_b%2Fcf C22_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:44231_uw_b%2Fcf, base_url=http://127.0.0.1:44231/uw_b/cf}
[junit4:junit4] 2> 69203 T309 C22 P44231 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 69204 T307 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 69204 T309 C22 P44231 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 69204 T309 C22 P44231 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 69205 T309 C22 P44231 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 69205 T231 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 69206 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 69206 T309 C22 P44231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 69206 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 69212 T267 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 69274 T231 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 69276 T231 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44218
[junit4:junit4] 2> 69277 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 69277 T231 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 69277 T231 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639
[junit4:junit4] 2> 69278 T231 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639/solr.xml
[junit4:junit4] 2> 69278 T231 oasc.CoreContainer.<init> New CoreContainer 1190966907
[junit4:junit4] 2> 69278 T231 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639/'
[junit4:junit4] 2> 69279 T231 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639/'
[junit4:junit4] 2> 69334 T231 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 69334 T231 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 69335 T231 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 69335 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 69335 T231 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 69335 T231 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 69336 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 69336 T231 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 69336 T231 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 69337 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 69341 T231 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 69341 T231 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53479/solr
[junit4:junit4] 2> 69342 T231 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 69342 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 69344 T321 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39bd5f67 name:ZooKeeperConnection Watcher:127.0.0.1:53479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 69344 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 69346 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 69350 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 69351 T323 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c1548bb name:ZooKeeperConnection Watcher:127.0.0.1:53479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 69352 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 69355 T231 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C23 name=collection1 org.apache.solr.core.SolrCore@7c20a9de url=http://127.0.0.1:44900/uw_b/cf/collection1 node=127.0.0.1:44900_uw_b%2Fcf C23_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:44900_uw_b%2Fcf, base_url=http://127.0.0.1:44900/uw_b/cf}
[junit4:junit4] 2> 69457 T292 C23 P44900 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:45675/uw_b/cf/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 69457 T292 C23 P44900 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf START replicas=[http://127.0.0.1:45675/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4] 2> 69458 T292 C23 P44900 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 69458 T292 C23 P44900 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 69458 T292 C23 P44900 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 69459 T292 C23 P44900 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 69459 T292 C23 P44900 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 69459 T292 C23 P44900 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:45675/uw_b/cf/collection1/. core=collection1
[junit4:junit4] 2> 69459 T292 C23 P44900 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C24 name=collection1 org.apache.solr.core.SolrCore@33ff417c url=http://127.0.0.1:45675/uw_b/cf/collection1 node=127.0.0.1:45675_uw_b%2Fcf C24_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:45675_uw_b%2Fcf, base_url=http://127.0.0.1:45675/uw_b/cf, leader=true}
[junit4:junit4] 2> 69464 T266 C24 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 69471 T268 C24 P45675 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 69473 T268 C24 P45675 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2179600a lockFactory=org.apache.lucene.store.NativeFSLockFactory@6cda313c),segFN=segments_1,generation=1}
[junit4:junit4] 2> 69473 T268 C24 P45675 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 69473 T268 C24 P45675 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 69474 T268 C24 P45675 oass.SolrIndexSearcher.<init> Opening Searcher@37c4d046 realtime
[junit4:junit4] 2> 69474 T268 C24 P45675 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 69474 T268 C24 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4] 2> 69490 T292 C23 P44900 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 69491 T292 C23 P44900 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4] 2> 69496 T292 C23 P44900 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 69499 T266 C24 P45675 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 69499 T266 C24 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 69500 T292 C23 P44900 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 69500 T292 C23 P44900 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 69501 T292 C23 P44900 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 69501 T292 C23 P44900 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 69502 T292 C23 P44900 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 69679 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 69681 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44231_uw_b%2Fcf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44231/uw_b/cf"}
[junit4:junit4] 2> 69685 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44900_uw_b%2Fcf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44900/uw_b/cf"}
[junit4:junit4] 2> 69690 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 69690 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 69691 T323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 69690 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 69690 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 69691 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 70212 T267 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 70213 T267 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node3&state=recovering&nodeName=127.0.0.1:44231_uw_b%252Fcf&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 70359 T231 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44218_uw_b%2Fcf
[junit4:junit4] 2> 70361 T231 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44218_uw_b%2Fcf
[junit4:junit4] 2> 70363 T323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 70363 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 70363 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 70363 T260 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 70364 T289 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 70364 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 70364 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 70366 T253 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 70366 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 70366 T323 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 70368 T306 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 70368 T273 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 70372 T325 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 70372 T325 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 70374 T325 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 71200 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 71201 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44218_uw_b%2Fcf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44218/uw_b/cf"}
[junit4:junit4] 2> 71201 T254 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 71202 T254 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 71206 T323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 71207 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 71207 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 71207 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 71207 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 71207 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 71374 T325 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 71374 T325 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639/collection1
[junit4:junit4] 2> 71375 T325 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 71375 T325 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 71376 T325 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 71377 T325 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639/collection1/'
[junit4:junit4] 2> 71378 T325 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639/collection1/lib/README' to classloader
[junit4:junit4] 2> 71378 T325 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 71410 T325 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 71452 T325 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 71454 T325 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 71458 T325 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 71840 T325 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 71847 T325 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 71849 T325 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 71870 T325 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 71875 T325 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 71881 T325 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 71882 T325 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 71883 T325 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 71883 T325 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 71884 T325 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 71885 T325 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 71885 T325 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 71885 T325 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty4/
[junit4:junit4] 2> 71885 T325 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@150858bb
[junit4:junit4] 2> 71886 T325 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty4
[junit4:junit4] 2> 71886 T325 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty4/index/
[junit4:junit4] 2> 71887 T325 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 71887 T325 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty4/index
[junit4:junit4] 2> 71889 T325 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2211c4af lockFactory=org.apache.lucene.store.NativeFSLockFactory@240c8343),segFN=segments_1,generation=1}
[junit4:junit4] 2> 71889 T325 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 71892 T325 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 71892 T325 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 71892 T325 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 71893 T325 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 71894 T325 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 71894 T325 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 71894 T325 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 71895 T325 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 71896 T325 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 71898 T325 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 71900 T325 oass.SolrIndexSearcher.<init> Opening Searcher@4863c568 main
[junit4:junit4] 2> 71901 T325 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 71901 T325 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 71909 T326 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4863c568 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 71912 T325 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 71912 T325 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44218/uw_b/cf collection:collection1 shard:shard1
[junit4:junit4] 2> 71917 T325 oasc.ZkController.register We are http://127.0.0.1:44218/uw_b/cf/collection1/ and leader is http://127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4] 2> 71917 T325 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44218/uw_b/cf
[junit4:junit4] 2> 71917 T325 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 71918 T325 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C25 name=collection1 org.apache.solr.core.SolrCore@345c4417 url=http://127.0.0.1:44218/uw_b/cf/collection1 node=127.0.0.1:44218_uw_b%2Fcf C25_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:44218_uw_b%2Fcf, base_url=http://127.0.0.1:44218/uw_b/cf}
[junit4:junit4] 2> 71918 T327 C25 P44218 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 71919 T327 C25 P44218 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 71919 T325 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 71919 T327 C25 P44218 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 71920 T327 C25 P44218 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 71921 T231 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 71921 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 71921 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 71923 T327 C25 P44218 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 71926 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 71928 T231 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30 for each attempt
[junit4:junit4] 2> 71928 T231 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
[junit4:junit4] 2> 71930 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 71930 T267 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> ASYNC NEW_CORE C26 name=collection1 org.apache.solr.core.SolrCore@4b2ea730 url=http://127.0.0.1:44231/uw_b/cf/collection1 node=127.0.0.1:44231_uw_b%2Fcf C26_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:44231_uw_b%2Fcf, base_url=http://127.0.0.1:44231/uw_b/cf}
[junit4:junit4] 2> 72214 T309 C26 P44231 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:45675/uw_b/cf/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 72215 T309 C26 P44231 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf START replicas=[http://127.0.0.1:45675/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4] 2> 72215 T309 C26 P44231 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 72216 T309 C26 P44231 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 72216 T309 C26 P44231 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 72216 T309 C26 P44231 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 72217 T309 C26 P44231 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 72217 T309 C26 P44231 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:45675/uw_b/cf/collection1/. core=collection1
[junit4:junit4] 2> 72217 T309 C26 P44231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C27 name=collection1 org.apache.solr.core.SolrCore@33ff417c url=http://127.0.0.1:45675/uw_b/cf/collection1 node=127.0.0.1:45675_uw_b%2Fcf C27_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:45675_uw_b%2Fcf, base_url=http://127.0.0.1:45675/uw_b/cf, leader=true}
[junit4:junit4] 2> 72218 T266 C27 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> ASYNC NEW_CORE C28 name=collection1 org.apache.solr.core.SolrCore@33ff417c url=http://127.0.0.1:45675/uw_b/cf/collection1 node=127.0.0.1:45675_uw_b%2Fcf C28_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:45675_uw_b%2Fcf, base_url=http://127.0.0.1:45675/uw_b/cf, leader=true}
[junit4:junit4] 2> 72237 T269 C28 P45675 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 72237 T269 C28 P45675 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 72238 T269 C28 P45675 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 72238 T269 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 2
[junit4:junit4] 2> ASYNC NEW_CORE C29 name=collection1 org.apache.solr.core.SolrCore@4b2ea730 url=http://127.0.0.1:44231/uw_b/cf/collection1 node=127.0.0.1:44231_uw_b%2Fcf C29_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:44231_uw_b%2Fcf, base_url=http://127.0.0.1:44231/uw_b/cf}
[junit4:junit4] 2> 72239 T309 C29 P44231 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 72240 T309 C29 P44231 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 72243 T266 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 72244 T309 C29 P44231 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 72244 T309 C29 P44231 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 72244 T309 C29 P44231 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 72245 T309 C29 P44231 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 72246 T309 C29 P44231 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 72714 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 72716 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44218_uw_b%2Fcf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44218/uw_b/cf"}
[junit4:junit4] 2> 72721 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44231_uw_b%2Fcf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44231/uw_b/cf"}
[junit4:junit4] 2> 72726 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 72726 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 72727 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 72726 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 72726 T323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 72726 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 72931 T267 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 72932 T267 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node4&state=recovering&nodeName=127.0.0.1:44218_uw_b%252Fcf&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4] 2> 72932 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 73934 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C30 name=collection1 org.apache.solr.core.SolrCore@345c4417 url=http://127.0.0.1:44218/uw_b/cf/collection1 node=127.0.0.1:44218_uw_b%2Fcf C30_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:44218_uw_b%2Fcf, base_url=http://127.0.0.1:44218/uw_b/cf}
[junit4:junit4] 2> 74933 T327 C30 P44218 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:45675/uw_b/cf/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 74933 T327 C30 P44218 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf START replicas=[http://127.0.0.1:45675/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4] 2> 74933 T327 C30 P44218 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 74934 T327 C30 P44218 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 74934 T327 C30 P44218 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 74934 T327 C30 P44218 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 74934 T327 C30 P44218 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 74934 T327 C30 P44218 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:45675/uw_b/cf/collection1/. core=collection1
[junit4:junit4] 2> 74935 T327 C30 P44218 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 74936 T269 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 74939 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 74941 T266 C28 P45675 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 74942 T266 C28 P45675 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 74942 T266 C28 P45675 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 74942 T266 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 1
[junit4:junit4] 2> 74943 T327 C30 P44218 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 74943 T327 C30 P44218 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 74946 T268 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 74947 T327 C30 P44218 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 74947 T327 C30 P44218 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 74947 T327 C30 P44218 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 74947 T327 C30 P44218 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 74948 T327 C30 P44218 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 75739 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 75740 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44218_uw_b%2Fcf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44218/uw_b/cf"}
[junit4:junit4] 2> 75746 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 75746 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 75746 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 75746 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 75746 T323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 75746 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 75942 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 75943 T231 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C31 name=collection1 org.apache.solr.core.SolrCore@667376df url=http://127.0.0.1:49202/uw_b/cf/collection1 node=127.0.0.1:49202_uw_b%2Fcf C31_STATE=coll:control_collection core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:49202_uw_b%2Fcf, base_url=http://127.0.0.1:49202/uw_b/cf, leader=true}
[junit4:junit4] 2> 75951 T249 C31 P49202 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 75953 T249 C31 P49202 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@211782bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@8504643),segFN=segments_1,generation=1}
[junit4:junit4] 2> 75954 T249 C31 P49202 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 75955 T249 C31 P49202 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 75955 T249 C31 P49202 oass.SolrIndexSearcher.<init> Opening Searcher@46bdc822 main
[junit4:junit4] 2> 75956 T249 C31 P49202 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 75957 T257 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@46bdc822 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 75957 T249 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> ASYNC NEW_CORE C32 name=collection1 org.apache.solr.core.SolrCore@7c20a9de url=http://127.0.0.1:44900/uw_b/cf/collection1 node=127.0.0.1:44900_uw_b%2Fcf C32_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:44900_uw_b%2Fcf, base_url=http://127.0.0.1:44900/uw_b/cf}
[junit4:junit4] 2> 75968 T285 C32 P44900 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:45675/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44900/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44231/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44218/uw_b/cf/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 75979 T282 C32 P44900 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 75983 T268 C28 P45675 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 75984 T268 C28 P45675 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> C29_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:44231_uw_b%2Fcf, base_url=http://127.0.0.1:44231/uw_b/cf}
[junit4:junit4] 2> 75984 T301 C29 P44231 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 75986 T268 C28 P45675 oass.SolrIndexSearcher.<init> Opening Searcher@668911a6 main
[junit4:junit4] 2> C30_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:44218_uw_b%2Fcf, base_url=http://127.0.0.1:44218/uw_b/cf}
[junit4:junit4] 2> 75987 T316 C30 P44218 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 75988 T301 C29 P44231 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3267b8c3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4389c038),segFN=segments_1,generation=1}
[junit4:junit4] 2> 75986 T282 C32 P44900 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@64f56312 lockFactory=org.apache.lucene.store.NativeFSLockFactory@50b7fbb8),segFN=segments_1,generation=1}
[junit4:junit4] 2> 75988 T275 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@668911a6 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 75987 T268 C28 P45675 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 75989 T316 C30 P44218 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2211c4af lockFactory=org.apache.lucene.store.NativeFSLockFactory@240c8343),segFN=segments_1,generation=1}
[junit4:junit4] 2> 75989 T282 C32 P44900 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 75989 T301 C29 P44231 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 75992 T282 C32 P44900 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 75992 T316 C30 P44218 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 75991 T268 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 76002 T316 C30 P44218 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 76008 T301 C29 P44231 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 76009 T282 C32 P44900 oass.SolrIndexSearcher.<init> Opening Searcher@17f7e334 main
[junit4:junit4] 2> 76011 T291 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@17f7e334 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 76012 T282 C32 P44900 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 76013 T282 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 34
[junit4:junit4] 2> 76014 T316 C30 P44218 oass.SolrIndexSearcher.<init> Opening Searcher@5a0ed63f main
[junit4:junit4] 2> 76016 T326 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5a0ed63f main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 76017 T316 C30 P44218 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 76017 T316 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 31
[junit4:junit4] 2> 76019 T301 C29 P44231 oass.SolrIndexSearcher.<init> Opening Searcher@39048c5d main
[junit4:junit4] 2> 76019 T301 C29 P44231 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 76020 T308 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@39048c5d main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 76021 T301 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 37
[junit4:junit4] 2> 76022 T285 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 54
[junit4:junit4] 2> 76024 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 76028 T266 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 76032 T283 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 76036 T300 C29 P44231 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 76040 T318 C30 P44218 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 78051 T247 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1439084734552473600)} 0 7
[junit4:junit4] 2> 78072 T300 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={update.distrib=FROMLEADER&_version_=-1439084734566105088&update.from=http://127.0.0.1:45675/uw_b/cf/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439084734566105088)} 0 1
[junit4:junit4] 2> 78072 T316 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={update.distrib=FROMLEADER&_version_=-1439084734566105088&update.from=http://127.0.0.1:45675/uw_b/cf/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439084734566105088)} 0 1
[junit4:junit4] 2> 78085 T285 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={update.distrib=FROMLEADER&_version_=-1439084734566105088&update.from=http://127.0.0.1:45675/uw_b/cf/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439084734566105088)} 0 2
[junit4:junit4] 2> 78086 T268 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1439084734566105088)} 0 27
[junit4:junit4] 2> 78087 T318 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 33
[junit4:junit4] 2> 78100 T249 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {add=[0 (1439084734600708096)]} 0 9
[junit4:junit4] 2> 78121 T319 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439084734620631040)]} 0 3
[junit4:junit4] 2> 78121 T284 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439084734620631040)]} 0 3
[junit4:junit4] 2> 78124 T302 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439084734620631040)]} 0 4
[junit4:junit4] 2> 78125 T266 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:44231/uw_b/cf/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1439084734620631040)]} 0 13
[junit4:junit4] 2> 78126 T301 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {add=[0]} 0 22
[junit4:junit4] 2> 78131 T248 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {add=[1 (1439084734640553984)]} 0 1
[junit4:junit4] 2> 78151 T317 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439084734651039744)]} 0 4
[junit4:junit4] 2> 78154 T283 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439084734651039744)]} 0 8
[junit4:junit4] 2> 78156 T299 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439084734651039744)]} 0 6
[junit4:junit4] 2> 78158 T267 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:44231/uw_b/cf/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1439084734651039744)]} 0 18
[junit4:junit4] 2> 78159 T300 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {add=[1]} 0 24
[junit4:junit4] 2> 78164 T246 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {add=[2 (1439084734675156992)]} 0 2
[junit4:junit4] 2> 78168 T268 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:44900/uw_b/cf/collection1/]
[junit4:junit4] 2> 78169 T268 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44900/uw_b/cf/collection1/ against:[http://127.0.0.1:44900/uw_b/cf/collection1/] result:true
[junit4:junit4] 2> 78169 T268 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44231/uw_b/cf/collection1/ against:[http://127.0.0.1:44900/uw_b/cf/collection1/] result:false
[junit4:junit4] 2> 78169 T268 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44218/uw_b/cf/collection1/ against:[http://127.0.0.1:44900/uw_b/cf/collection1/] result:false
[junit4:junit4] 2> 78178 T316 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:44900/uw_b/cf/collection1/&wt=javabin&version=2} {add=[2 (1439084734681448448)]} 0 4
[junit4:junit4] 2> 78180 T302 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:44900/uw_b/cf/collection1/&wt=javabin&version=2} {add=[2 (1439084734681448448)]} 0 4
[junit4:junit4] 2> 78181 T268 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={test.distrib.skip.servers=http://127.0.0.1:44900/uw_b/cf/collection1/&wt=javabin&version=2} {add=[2 (1439084734681448448)]} 0 13
[junit4:junit4] 2> 78187 T247 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {add=[3 (1439084734698225664)]} 0 2
[junit4:junit4] 2> 78192 T269 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:44900/uw_b/cf/collection1/, http://127.0.0.1:44231/uw_b/cf/collection1/]
[junit4:junit4] 2> 78193 T269 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44900/uw_b/cf/collection1/ against:[http://127.0.0.1:44900/uw_b/cf/collection1/, http://127.0.0.1:44231/uw_b/cf/collection1/] result:true
[junit4:junit4] 2> 78193 T269 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44231/uw_b/cf/collection1/ against:[http://127.0.0.1:44900/uw_b/cf/collection1/, http://127.0.0.1:44231/uw_b/cf/collection1/] result:true
[junit4:junit4] 2> 78194 T269 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44218/uw_b/cf/collection1/ against:[http://127.0.0.1:44900/uw_b/cf/collection1/, http://127.0.0.1:44231/uw_b/cf/collection1/] result:false
[junit4:junit4] 2> 78201 T318 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:44900/uw_b/cf/collection1/&wt=javabin&version=2} {add=[3 (1439084734707662848)]} 0 2
[junit4:junit4] 2> 78203 T269 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={test.distrib.skip.servers=http://127.0.0.1:44900/uw_b/cf/collection1/&test.distrib.skip.servers=http://127.0.0.1:44231/uw_b/cf/collection1/&wt=javabin&version=2} {add=[3 (1439084734707662848)]} 0 12
[junit4:junit4] 2> 78206 T249 C31 P49202 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 78221 T249 C31 P49202 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@211782bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@8504643),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@211782bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@8504643),segFN=segments_2,generation=2}
[junit4:junit4] 2> 78222 T249 C31 P49202 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 78230 T249 C31 P49202 oass.SolrIndexSearcher.<init> Opening Searcher@3b5125e8 main
[junit4:junit4] 2> 78231 T249 C31 P49202 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 78232 T257 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3b5125e8 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 78233 T249 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 27
[junit4:junit4] 2> 78237 T266 C28 P45675 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:45675/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44900/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44231/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44218/uw_b/cf/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 78241 T301 C29 P44231 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 78241 T282 C32 P44900 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 78241 T267 C28 P45675 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 78242 T319 C30 P44218 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 78256 T301 C29 P44231 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3267b8c3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4389c038),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3267b8c3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4389c038),segFN=segments_2,generation=2}
[junit4:junit4] 2> 78256 T282 C32 P44900 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@64f56312 lockFactory=org.apache.lucene.store.NativeFSLockFactory@50b7fbb8),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@64f56312 lockFactory=org.apache.lucene.store.NativeFSLockFactory@50b7fbb8),segFN=segments_2,generation=2}
[junit4:junit4] 2> 78257 T267 C28 P45675 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2179600a lockFactory=org.apache.lucene.store.NativeFSLockFactory@6cda313c),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2179600a lockFactory=org.apache.lucene.store.NativeFSLockFactory@6cda313c),segFN=segments_2,generation=2}
[junit4:junit4] 2> 78259 T267 C28 P45675 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 78256 T301 C29 P44231 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 78258 T282 C32 P44900 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 78268 T267 C28 P45675 oass.SolrIndexSearcher.<init> Opening Searcher@474a1d0 main
[junit4:junit4] 2> 78268 T319 C30 P44218 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2211c4af lockFactory=org.apache.lucene.store.NativeFSLockFactory@240c8343),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2211c4af lockFactory=org.apache.lucene.store.NativeFSLockFactory@240c8343),segFN=segments_2,generation=2}
[junit4:junit4] 2> 78272 T267 C28 P45675 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 78272 T319 C30 P44218 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 78274 T275 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@474a1d0 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 78275 T267 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 34
[junit4:junit4] 2> 78276 T282 C32 P44900 oass.SolrIndexSearcher.<init> Opening Searcher@79382710 main
[junit4:junit4] 2> 78276 T319 C30 P44218 oass.SolrIndexSearcher.<init> Opening Searcher@2d153029 main
[junit4:junit4] 2> 78277 T282 C32 P44900 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 78278 T291 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79382710 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C2)}
[junit4:junit4] 2> 78279 T282 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 38
[junit4:junit4] 2> 78277 T301 C29 P44231 oass.SolrIndexSearcher.<init> Opening Searcher@338353fd main
[junit4:junit4] 2> 78277 T319 C30 P44218 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 78281 T308 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@338353fd main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C3)}
[junit4:junit4] 2> 78280 T301 C29 P44231 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 78280 T326 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2d153029 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 78284 T301 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 44
[junit4:junit4] 2> 78285 T319 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 43
[junit4:junit4] 2> 78286 T266 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 49
[junit4:junit4] 2> 78287 T231 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):330
[junit4:junit4] 2> 78288 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 78289 T231 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 78292 T268 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 78296 T285 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0
[junit4:junit4] 2> 78300 T299 C29 P44231 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=3 status=0 QTime=0
[junit4:junit4] 2> 78304 T317 C30 P44218 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 78307 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 78314 T300 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4] 2> 78315 T300 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 78323 T267 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4] 2> 78323 T267 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4] 2> 78324 T267 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45675/uw_b/cf START replicas=[http://127.0.0.1:44900/uw_b/cf/collection1/, http://127.0.0.1:44231/uw_b/cf/collection1/, http://127.0.0.1:44218/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4] 2> 78328 T284 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 78328 T316 C30 P44218 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 78328 T302 C29 P44231 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 78329 T267 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45675/uw_b/cf Received 3 versions from 127.0.0.1:44900/uw_b/cf/collection1/
[junit4:junit4] 2> 78330 T267 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45675/uw_b/cf Our versions are newer. ourLowThreshold=1439084734566105088 otherHigh=1439084734651039744
[junit4:junit4] 2> 78330 T267 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45675/uw_b/cf Received 5 versions from 127.0.0.1:44218/uw_b/cf/collection1/
[junit4:junit4] 2> 78330 T267 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45675/uw_b/cf Our versions are newer. ourLowThreshold=1439084734566105088 otherHigh=1439084734681448448
[junit4:junit4] 2> 78330 T267 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45675/uw_b/cf Received 4 versions from 127.0.0.1:44231/uw_b/cf/collection1/
[junit4:junit4] 2> 78331 T267 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45675/uw_b/cf Our versions are newer. ourLowThreshold=1439084734566105088 otherHigh=1439084734681448448
[junit4:junit4] 2> 78331 T267 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45675/uw_b/cf DONE. sync succeeded
[junit4:junit4] 2> 78331 T267 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 78331 T267 oasc.SyncStrategy.syncToMe http://127.0.0.1:45675/uw_b/cf/collection1/: try and ask http://127.0.0.1:44900/uw_b/cf/collection1/ to sync
[junit4:junit4] 2> 78332 T267 oasc.SyncStrategy.syncToMe http://127.0.0.1:45675/uw_b/cf/collection1/: try and ask http://127.0.0.1:44231/uw_b/cf/collection1/ to sync
[junit4:junit4] 2> 78333 T267 oasc.SyncStrategy.syncToMe http://127.0.0.1:45675/uw_b/cf/collection1/: try and ask http://127.0.0.1:44218/uw_b/cf/collection1/ to sync
[junit4:junit4] 2> 78336 T283 C32 P44900 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf START replicas=[http://127.0.0.1:45675/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4] 2> 78336 T299 C29 P44231 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf START replicas=[http://127.0.0.1:45675/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4] 2> 78339 T318 C30 P44218 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf START replicas=[http://127.0.0.1:45675/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4] 2> 78339 T268 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 78340 T283 C32 P44900 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf Received 5 versions from 127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4] 2> 78341 T269 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 78341 T283 C32 P44900 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf Requesting updates from 127.0.0.1:45675/uw_b/cf/collection1/n=2 versions=[1439084734707662848, 1439084734681448448]
[junit4:junit4] 2> 78342 T299 C29 P44231 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf Received 5 versions from 127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4] 2> 78342 T299 C29 P44231 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf Requesting updates from 127.0.0.1:45675/uw_b/cf/collection1/n=1 versions=[1439084734707662848]
[junit4:junit4] 2> 78343 T266 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 78344 T318 C30 P44218 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf Received 5 versions from 127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4] 2> 78344 T318 C30 P44218 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf Our versions are newer. ourLowThreshold=1439084734566105088 otherHigh=1439084734681448448
[junit4:junit4] 2> 78345 T318 C30 P44218 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf DONE. sync succeeded
[junit4:junit4] 2> 78345 T318 C30 P44218 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={sync=http://127.0.0.1:45675/uw_b/cf/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=6
[junit4:junit4] 2> 78346 T268 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={distrib=false&getUpdates=1439084734707662848,1439084734681448448&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 78346 T267 oasc.SyncStrategy.syncToMe http://127.0.0.1:45675/uw_b/cf/collection1/: sync completed with http://127.0.0.1:44218/uw_b/cf/collection1/
[junit4:junit4] 2> 78348 T269 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={distrib=false&getUpdates=1439084734707662848&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 78351 T299 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] {add=[3 (1439084734707662848)]} 0 2
[junit4:junit4] 2> 78352 T299 C29 P44231 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf DONE. sync succeeded
[junit4:junit4] 2> 78352 T299 C29 P44231 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={sync=http://127.0.0.1:45675/uw_b/cf/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=16
[junit4:junit4] 2> 78353 T267 oasc.SyncStrategy.syncToMe http://127.0.0.1:45675/uw_b/cf/collection1/: sync completed with http://127.0.0.1:44231/uw_b/cf/collection1/
[junit4:junit4] 2> 78354 T283 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1439084734681448448), 3 (1439084734707662848)]} 0 8
[junit4:junit4] 2> 78354 T283 C32 P44900 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf DONE. sync succeeded
[junit4:junit4] 2> 78355 T283 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={sync=http://127.0.0.1:45675/uw_b/cf/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=19
[junit4:junit4] 2> 78356 T267 oasc.SyncStrategy.syncToMe http://127.0.0.1:45675/uw_b/cf/collection1/: sync completed with http://127.0.0.1:44900/uw_b/cf/collection1/
[junit4:junit4] 2> 78356 T267 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=33
[junit4:junit4] 2> 78357 T300 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=45
[junit4:junit4] 2> 78358 T231 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 78358 T231 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 78359 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 78361 T231 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 78364 T246 C31 P49202 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 78365 T246 C31 P49202 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 78365 T246 C31 P49202 oass.SolrIndexSearcher.<init> Opening Searcher@3a5edc81 main
[junit4:junit4] 2> 78366 T246 C31 P49202 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 78368 T257 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3a5edc81 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 78368 T246 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4] 2> 78372 T266 C28 P45675 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:45675/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44900/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44231/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44218/uw_b/cf/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 78376 T268 C28 P45675 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 78376 T282 C32 P44900 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 78376 T319 C30 P44218 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 78376 T302 C29 P44231 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 78377 T319 C30 P44218 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 78376 T268 C28 P45675 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 78379 T268 C28 P45675 oass.SolrIndexSearcher.<init> Opening Searcher@4d5c9173 main
[junit4:junit4] 2> 78380 T268 C28 P45675 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 78382 T275 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4d5c9173 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 78379 T319 C30 P44218 oass.SolrIndexSearcher.<init> Opening Searcher@22be1af6 main
[junit4:junit4] 2> 78382 T268 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> 78383 T319 C30 P44218 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 78383 T282 C32 P44900 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@64f56312 lockFactory=org.apache.lucene.store.NativeFSLockFactory@50b7fbb8),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@64f56312 lockFactory=org.apache.lucene.store.NativeFSLockFactory@50b7fbb8),segFN=segments_3,generation=3}
[junit4:junit4] 2> 78384 T282 C32 P44900 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 78384 T326 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@22be1af6 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 78389 T319 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 13
[junit4:junit4] 2> 78390 T282 C32 P44900 oass.SolrIndexSearcher.<init> Opening Searcher@43b4e3fd main
[junit4:junit4] 2> 78391 T302 C29 P44231 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3267b8c3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4389c038),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3267b8c3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4389c038),segFN=segments_3,generation=3}
[junit4:junit4] 2> 78392 T302 C29 P44231 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 78392 T291 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@43b4e3fd main{StandardDirectoryReader(segments_3:5:nrt _0(5.0):C2 _1(5.0):C2)}
[junit4:junit4] 2> 78394 T282 C32 P44900 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 78394 T302 C29 P44231 oass.SolrIndexSearcher.<init> Opening Searcher@5a77cf58 main
[junit4:junit4] 2> 78395 T282 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 19
[junit4:junit4] 2> 78396 T308 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5a77cf58 main{StandardDirectoryReader(segments_3:5:nrt _0(5.0):C3 _1(5.0):C1)}
[junit4:junit4] 2> 78397 T302 C29 P44231 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 78398 T302 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 22
[junit4:junit4] 2> 78399 T266 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 27
[junit4:junit4] 2> 78400 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 78404 T269 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 78407 T285 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 78410 T301 C29 P44231 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 78414 T317 C30 P44218 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 80416 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> check const of shard1
[junit4:junit4] 2> client0
[junit4:junit4] 2> PROPS:core_node1:{"state":"active","core":"collection1","node_name":"127.0.0.1:45675_uw_b%2Fcf","base_url":"http://127.0.0.1:45675/uw_b/cf","leader":"true"}
[junit4:junit4] 2> 80419 T266 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client1
[junit4:junit4] 2> PROPS:core_node2:{"state":"active","core":"collection1","node_name":"127.0.0.1:44900_uw_b%2Fcf","base_url":"http://127.0.0.1:44900/uw_b/cf"}
[junit4:junit4] 2> 80422 T285 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client2
[junit4:junit4] 2> PROPS:core_node3:{"state":"active","core":"collection1","node_name":"127.0.0.1:44231_uw_b%2Fcf","base_url":"http://127.0.0.1:44231/uw_b/cf"}
[junit4:junit4] 2> 80425 T302 C29 P44231 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client3
[junit4:junit4] 2> PROPS:core_node4:{"state":"active","core":"collection1","node_name":"127.0.0.1:44218_uw_b%2Fcf","base_url":"http://127.0.0.1:44218/uw_b/cf"}
[junit4:junit4] 2> 80428 T319 C30 P44218 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> 80431 T283 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={q=*:*&wt=javabin&version=2} hits=4 status=0 QTime=0
[junit4:junit4] 2> 80439 T249 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {add=[4 (1439084737058570240)]} 0 3
[junit4:junit4] 2> 80444 T269 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:44218/uw_b/cf/collection1/]
[junit4:junit4] 2> 80445 T269 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44900/uw_b/cf/collection1/ against:[http://127.0.0.1:44218/uw_b/cf/collection1/] result:false
[junit4:junit4] 2> 80445 T269 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44231/uw_b/cf/collection1/ against:[http://127.0.0.1:44218/uw_b/cf/collection1/] result:false
[junit4:junit4] 2> 80445 T269 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44218/uw_b/cf/collection1/ against:[http://127.0.0.1:44218/uw_b/cf/collection1/] result:true
[junit4:junit4] 2> 80452 T301 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:44218/uw_b/cf/collection1/&wt=javabin&version=2} {add=[4 (1439084737069056000)]} 0 2
[junit4:junit4] 2> 80453 T282 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:44218/uw_b/cf/collection1/&wt=javabin&version=2} {add=[4 (1439084737069056000)]} 0 3
[junit4:junit4] 2> 80454 T269 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:44218/uw_b/cf/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:44218/uw_b/cf/collection1/&wt=javabin&version=2} {add=[4 (1439084737069056000)]} 0 10
[junit4:junit4] 2> 80454 T317 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={test.distrib.skip.servers=http://127.0.0.1:44218/uw_b/cf/collection1/&wt=javabin&version=2} {add=[4]} 0 12
[junit4:junit4] 2> 80455 T231 oasc.ChaosMonkey.monkeyLog monkey: kill shard! 45675
[junit4:junit4] 2> 80477 T231 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=2095146885
[junit4:junit4] 2> 81774 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 81776 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:45675_uw_b%2Fcf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45675/uw_b/cf"}
[junit4:junit4] 2> 81776 T254 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4:junit4] 2> 81780 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 81780 T323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 81780 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 81780 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 81780 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 81780 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 83480 T231 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 83480 T231 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 83482 T231 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@33ff417c
[junit4:junit4] 2> 83488 T231 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=5,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 83489 T231 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 83489 T231 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 83489 T231 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> C28_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:45675_uw_b%2Fcf, base_url=http://127.0.0.1:45675/uw_b/cf, leader=true}
[junit4:junit4] 2> 83494 T231 C28 P45675 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2179600a lockFactory=org.apache.lucene.store.NativeFSLockFactory@6cda313c),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2179600a lockFactory=org.apache.lucene.store.NativeFSLockFactory@6cda313c),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2179600a lockFactory=org.apache.lucene.store.NativeFSLockFactory@6cda313c),segFN=segments_3,generation=3}
[junit4:junit4] 2> 83495 T231 C28 P45675 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 83496 T231 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 83497 T231 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 83497 T231 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1;done=false>>]
[junit4:junit4] 2> 83497 T231 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1
[junit4:junit4] 2> 83498 T231 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1/index;done=false>>]
[junit4:junit4] 2> 83498 T231 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1/index
[junit4:junit4] 2> 83500 T323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 83500 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 83500 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 83501 T253 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 83501 T273 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 83502 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 83501 T231 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/uw_b/cf,null}
[junit4:junit4] 2> 83502 T273 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 83504 T260 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 83505 T289 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 83505 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 83505 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 83509 T323 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 83509 T306 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 83512 T289 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4] 2> 83515 T289 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 83515 T289 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 83515 T289 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 83515 T289 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44900/uw_b/cf/collection1/
[junit4:junit4] 2> 83515 T289 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf START replicas=[http://127.0.0.1:44231/uw_b/cf/collection1/, http://127.0.0.1:44218/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4] 2> 83519 T319 C30 P44218 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 83519 T302 C29 P44231 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 83520 T289 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf Received 5 versions from 127.0.0.1:44218/uw_b/cf/collection1/
[junit4:junit4] 2> 83520 T289 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf Our versions are newer. ourLowThreshold=1439084734620631040 otherHigh=1439084734681448448
[junit4:junit4] 2> 83521 T289 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf Received 6 versions from 127.0.0.1:44231/uw_b/cf/collection1/
[junit4:junit4] 2> 83521 T289 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf Our versions are newer. ourLowThreshold=1439084734620631040 otherHigh=1439084734707662848
[junit4:junit4] 2> 83521 T289 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf DONE. sync succeeded
[junit4:junit4] 2> 83522 T289 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 83522 T289 oasc.SyncStrategy.syncToMe http://127.0.0.1:44900/uw_b/cf/collection1/: try and ask http://127.0.0.1:44231/uw_b/cf/collection1/ to sync
[junit4:junit4] 2> 83522 T289 oasc.SyncStrategy.syncToMe http://127.0.0.1:44900/uw_b/cf/collection1/: try and ask http://127.0.0.1:44218/uw_b/cf/collection1/ to sync
[junit4:junit4] 2> 83525 T301 C29 P44231 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf START replicas=[http://127.0.0.1:44900/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4] 2> 83525 T318 C30 P44218 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf START replicas=[http://127.0.0.1:44900/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4] 2> 83528 T284 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 83529 T318 C30 P44218 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf Received 6 versions from 127.0.0.1:44900/uw_b/cf/collection1/
[junit4:junit4] 2> 83529 T318 C30 P44218 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf Requesting updates from 127.0.0.1:44900/uw_b/cf/collection1/n=1 versions=[1439084737069056000]
[junit4:junit4] 2> 83530 T285 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 83530 T301 C29 P44231 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf Received 6 versions from 127.0.0.1:44900/uw_b/cf/collection1/
[junit4:junit4] 2> 83531 T301 C29 P44231 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf Our versions are newer. ourLowThreshold=1439084734620631040 otherHigh=1439084734707662848
[junit4:junit4] 2> 83531 T301 C29 P44231 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf DONE. sync succeeded
[junit4:junit4] 2> 83532 T283 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={distrib=false&getUpdates=1439084737069056000&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 83532 T301 C29 P44231 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={sync=http://127.0.0.1:44900/uw_b/cf/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=7
[junit4:junit4] 2> 83533 T289 oasc.SyncStrategy.syncToMe http://127.0.0.1:44900/uw_b/cf/collection1/: sync completed with http://127.0.0.1:44231/uw_b/cf/collection1/
[junit4:junit4] 2> 83535 T318 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] {add=[4 (1439084737069056000)]} 0 3
[junit4:junit4] 2> 83535 T318 C30 P44218 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf DONE. sync succeeded
[junit4:junit4] 2> 83536 T318 C30 P44218 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={sync=http://127.0.0.1:44900/uw_b/cf/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=11
[junit4:junit4] 2> 83537 T289 oasc.SyncStrategy.syncToMe http://127.0.0.1:44900/uw_b/cf/collection1/: sync completed with http://127.0.0.1:44218/uw_b/cf/collection1/
[junit4:junit4] 2> 83537 T289 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44900/uw_b/cf/collection1/ shard1
[junit4:junit4] 2> 83537 T289 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 83554 T231 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 45675
[junit4:junit4] 2> 84789 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 84790 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45675_uw_b%2Fcf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45675/uw_b/cf"}
[junit4:junit4] 2> 84808 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 84808 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 84808 T323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 84808 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 84808 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 85555 T231 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 90 for each attempt
[junit4:junit4] 2> 85555 T231 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):90
[junit4:junit4] 2> 85556 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 85558 T231 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 85560 T247 C31 P49202 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 85564 T247 C31 P49202 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@211782bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@8504643),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@211782bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@8504643),segFN=segments_3,generation=3}
[junit4:junit4] 2> 85565 T247 C31 P49202 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 85566 T247 C31 P49202 oass.SolrIndexSearcher.<init> Opening Searcher@579abafe main
[junit4:junit4] 2> 85567 T247 C31 P49202 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 85569 T257 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@579abafe main{StandardDirectoryReader(segments_3:5:nrt _0(5.0):C4 _1(5.0):C1)}
[junit4:junit4] 2> 85569 T247 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4] 2> 85573 T319 C30 P44218 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:44900/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44231/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44218/uw_b/cf/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> C32_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:44900_uw_b%2Fcf, base_url=http://127.0.0.1:44900/uw_b/cf, leader=true}
[junit4:junit4] 2> 85576 T285 C32 P44900 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 85576 T301 C29 P44231 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 85576 T316 C30 P44218 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 85581 T301 C29 P44231 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3267b8c3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4389c038),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3267b8c3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4389c038),segFN=segments_4,generation=4}
[junit4:junit4] 2> 85581 T285 C32 P44900 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@64f56312 lockFactory=org.apache.lucene.store.NativeFSLockFactory@50b7fbb8),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@64f56312 lockFactory=org.apache.lucene.store.NativeFSLockFactory@50b7fbb8),segFN=segments_4,generation=4}
[junit4:junit4] 2> 85581 T285 C32 P44900 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 85581 T301 C29 P44231 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 85582 T316 C30 P44218 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirW
[...truncated too long message...]
{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{
[junit4:junit4] 1> "core_node1":{
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:45675_uw_b%2Fcf",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:45675/uw_b/cf"},
[junit4:junit4] 1> "core_node2":{
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:44900_uw_b%2Fcf",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:44900/uw_b/cf"},
[junit4:junit4] 1> "core_node3":{
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:44231_uw_b%2Fcf",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:44231/uw_b/cf",
[junit4:junit4] 1> "leader":"true"},
[junit4:junit4] 1> "core_node4":{
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:44218_uw_b%2Fcf",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:44218/uw_b/cf"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"core_node1":{
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:49202_uw_b%2Fcf",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:49202/uw_b/cf",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:44231_uw_b%2Fcf (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:45675_uw_b%2Fcf (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:44218_uw_b%2Fcf (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:49202_uw_b%2Fcf (0)
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/collections (2)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89942794542055436-core_node4-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89942794542055438-core_node1-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89942794542055434-core_node3-n_0000000002 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (1)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:44231_uw_b%2Fcf",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:44231/uw_b/cf"}
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89942794542055427-core_node1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:49202_uw_b%2Fcf",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:49202/uw_b/cf"}
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/election (4)
[junit4:junit4] 1> /solr/overseer_elect/election/89942794542055438-127.0.0.1:45675_uw_b%2Fcf-n_0000000005 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89942794542055436-127.0.0.1:44218_uw_b%2Fcf-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89942794542055427-127.0.0.1:49202_uw_b%2Fcf-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89942794542055434-127.0.0.1:44231_uw_b%2Fcf-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89942794542055427-127.0.0.1:49202_uw_b%2Fcf-n_0000000000"}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=4C71556C638378C7 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=de_DE -Dtests.timezone=US/Alaska -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 99.6s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:45675/uw_b/cf/collection1lastClient and got 261 from http://127.0.0.1:44231/uw_b/cf/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([4C71556C638378C7:CD97DB7414DC18FB]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1045)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:237)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 156798 T231 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 99634 T230 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
[junit4:junit4] 2> 156889 T306 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 156889 T306 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 157977 T323 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 157978 T323 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=de_DE, timezone=US/Alaska
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=8,threads=1,free=68633968,total=131072000
[junit4:junit4] 2> NOTE: All tests run in this JVM: [ReturnFieldsTest, ClusterStateUpdateTest, DirectUpdateHandlerOptimizeTest, PeerSyncTest, TestSchemaNameResource, TestSort, SystemInfoHandlerTest, IndexSchemaTest, PrimitiveFieldTypeTest, TestPropInjectDefaults, BlockCacheTest, TestSurroundQueryParser, BasicFunctionalityTest, TestLFUCache, TestMaxScoreQueryParser, TestManagedSchemaFieldResource, SyncSliceTest]
[junit4:junit4] Completed on J0 in 100.92s, 1 test, 1 failure <<< FAILURES!
[...truncated 821 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:190: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:443: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1245: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:889: There were test failures: 312 suites, 1316 tests, 1 failure, 20 ignored (13 assumptions)
Total time: 39 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_25 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_25) - Build #
6348 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6348/
Java: 32bit/jdk1.7.0_25 -client -XX:+UseSerialGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest: 1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:
1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
at __randomizedtesting.SeedInfo.seed([7BB20C2B3A27495C]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
at __randomizedtesting.SeedInfo.seed([7BB20C2B3A27495C]:0)
Build Log:
[...truncated 9507 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 612255 T2992 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /x_/rw
[junit4:junit4] 2> 612266 T2992 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1372413435935
[junit4:junit4] 2> 612267 T2992 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 612267 T2993 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 612367 T2992 oasc.ZkTestServer.run start zk server on port:51741
[junit4:junit4] 2> 612368 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 612577 T2999 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c24fe name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 612578 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 612578 T2992 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 612583 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 612584 T3001 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f0fc30 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 612584 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 612584 T2992 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 612586 T2992 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 612588 T2992 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 612589 T2992 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 612591 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 612592 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 612595 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 612595 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 612598 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 612598 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 612600 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 612600 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 612602 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 612603 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 612605 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 612605 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 612607 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 612607 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 612609 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 612609 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 612611 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 612611 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 612709 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 612723 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45695
[junit4:junit4] 2> 612724 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 612724 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 612725 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281
[junit4:junit4] 2> 612725 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281/solr.xml
[junit4:junit4] 2> 612725 T2992 oasc.CoreContainer.<init> New CoreContainer 16159686
[junit4:junit4] 2> 612726 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281/'
[junit4:junit4] 2> 612726 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281/'
[junit4:junit4] 2> 612792 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 612792 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 612793 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 612793 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 612793 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 612794 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 612794 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 612794 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 612795 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 612795 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 612798 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 612799 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4] 2> 612799 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 612800 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 612801 T3012 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14f3ce8 name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 612801 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 612803 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 612806 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 612807 T3014 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@eac7a7 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 612807 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 612809 T2992 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 612811 T2992 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 612813 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 612815 T2992 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45695_x_%2Frw
[junit4:junit4] 2> 612815 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45695_x_%2Frw
[junit4:junit4] 2> 612818 T2992 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 612821 T2992 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 612823 T2992 oasc.Overseer.start Overseer (id=89942486937567235-127.0.0.1:45695_x_%2Frw-n_0000000000) starting
[junit4:junit4] 2> 612826 T2992 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 612828 T3016 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 612829 T2992 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 612830 T2992 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 612832 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 612834 T3015 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 612836 T3017 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 612837 T3017 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 612838 T3017 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 614337 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 614338 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45695_x_%2Frw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45695/x_/rw"}
[junit4:junit4] 2> 614338 T3015 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 614338 T3015 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 614341 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 614838 T3017 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 614838 T3017 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281/collection1
[junit4:junit4] 2> 614839 T3017 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 614839 T3017 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 614840 T3017 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 614841 T3017 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281/collection1/'
[junit4:junit4] 2> 614841 T3017 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281/collection1/lib/README' to classloader
[junit4:junit4] 2> 614842 T3017 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 614870 T3017 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 614907 T3017 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 614909 T3017 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 614913 T3017 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 615303 T3017 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 615309 T3017 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 615312 T3017 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 615321 T3017 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 615324 T3017 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 615327 T3017 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 615327 T3017 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 615328 T3017 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 615328 T3017 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 615329 T3017 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 615329 T3017 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 615329 T3017 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 615329 T3017 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/control/data/
[junit4:junit4] 2> 615329 T3017 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff291e
[junit4:junit4] 2> 615330 T3017 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/control/data
[junit4:junit4] 2> 615330 T3017 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/control/data/index/
[junit4:junit4] 2> 615331 T3017 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 615331 T3017 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/control/data/index
[junit4:junit4] 2> 615334 T3017 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 615334 T3017 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 615335 T3017 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 615336 T3017 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 615336 T3017 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 615337 T3017 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 615337 T3017 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 615337 T3017 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 615337 T3017 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 615338 T3017 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 615338 T3017 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 615340 T3017 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 615343 T3017 oass.SolrIndexSearcher.<init> Opening Searcher@1c81327 main
[junit4:junit4] 2> 615344 T3017 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 615344 T3017 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 615347 T3018 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c81327 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 615349 T3017 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 615349 T3017 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45695/x_/rw collection:control_collection shard:shard1
[junit4:junit4] 2> 615350 T3017 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 615354 T3017 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4] 2> 615356 T3017 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 615356 T3017 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 615356 T3017 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45695/x_/rw/collection1/
[junit4:junit4] 2> 615356 T3017 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 615357 T3017 oasc.SyncStrategy.syncToMe http://127.0.0.1:45695/x_/rw/collection1/ has no replicas
[junit4:junit4] 2> 615357 T3017 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45695/x_/rw/collection1/ shard1
[junit4:junit4] 2> 615357 T3017 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 615844 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 615850 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 615862 T3017 oasc.ZkController.register We are http://127.0.0.1:45695/x_/rw/collection1/ and leader is http://127.0.0.1:45695/x_/rw/collection1/
[junit4:junit4] 2> 615863 T3017 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45695/x_/rw
[junit4:junit4] 2> 615863 T3017 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 615863 T3017 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 615863 T3017 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 615866 T3017 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 615867 T2992 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 615867 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 615868 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 615871 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 615873 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 615874 T3021 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1436fa2 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 615874 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 615876 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 615877 T2992 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 615935 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 615937 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33049
[junit4:junit4] 2> 615938 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 615938 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 615938 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545
[junit4:junit4] 2> 615939 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545/solr.xml
[junit4:junit4] 2> 615939 T2992 oasc.CoreContainer.<init> New CoreContainer 22280971
[junit4:junit4] 2> 615939 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545/'
[junit4:junit4] 2> 615940 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545/'
[junit4:junit4] 2> 616002 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 616002 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 616002 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 616003 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 616003 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 616003 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 616003 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 616004 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 616004 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 616004 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 616007 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 616008 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4] 2> 616008 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 616009 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 616010 T3032 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e1661c name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 616011 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 616013 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 616015 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 616016 T3034 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14b1806 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 616017 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 616019 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 617022 T2992 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33049_x_%2Frw
[junit4:junit4] 2> 617023 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33049_x_%2Frw
[junit4:junit4] 2> 617026 T3014 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 617026 T3021 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 617026 T3034 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 617027 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 617031 T3035 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 617032 T3035 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 617033 T3035 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 617354 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 617354 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:45695_x_%2Frw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45695/x_/rw"}
[junit4:junit4] 2> 617357 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33049_x_%2Frw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33049/x_/rw"}
[junit4:junit4] 2> 617358 T3015 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 617358 T3015 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 617361 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 617361 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 617361 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 618033 T3035 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 618034 T3035 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545/collection1
[junit4:junit4] 2> 618034 T3035 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 618035 T3035 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 618036 T3035 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 618038 T3035 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545/collection1/'
[junit4:junit4] 2> 618039 T3035 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545/collection1/lib/README' to classloader
[junit4:junit4] 2> 618039 T3035 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 618085 T3035 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 618122 T3035 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 618123 T3035 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 618127 T3035 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 618550 T3035 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 618556 T3035 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 618559 T3035 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 618569 T3035 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 618572 T3035 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 618575 T3035 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 618576 T3035 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 618576 T3035 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 618576 T3035 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 618577 T3035 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 618577 T3035 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 618578 T3035 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 618578 T3035 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty1/
[junit4:junit4] 2> 618578 T3035 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff291e
[junit4:junit4] 2> 618579 T3035 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty1
[junit4:junit4] 2> 618579 T3035 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty1/index/
[junit4:junit4] 2> 618579 T3035 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 618580 T3035 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty1/index
[junit4:junit4] 2> 618582 T3035 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 618583 T3035 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 618584 T3035 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 618584 T3035 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 618585 T3035 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 618586 T3035 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 618586 T3035 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 618586 T3035 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 618587 T3035 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 618587 T3035 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 618587 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 618590 T3035 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 618593 T3035 oass.SolrIndexSearcher.<init> Opening Searcher@18a64f3 main
[junit4:junit4] 2> 618593 T3035 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 618593 T3035 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 618602 T3036 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@18a64f3 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 618605 T3035 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 618605 T3035 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33049/x_/rw collection:collection1 shard:shard1
[junit4:junit4] 2> 618606 T3035 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 618612 T3035 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4] 2> 618614 T3035 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 618614 T3035 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 618614 T3035 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33049/x_/rw/collection1/
[junit4:junit4] 2> 618615 T3035 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 618615 T3035 oasc.SyncStrategy.syncToMe http://127.0.0.1:33049/x_/rw/collection1/ has no replicas
[junit4:junit4] 2> 618615 T3035 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33049/x_/rw/collection1/ shard1
[junit4:junit4] 2> 618616 T3035 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 618866 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 618872 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 618872 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 618872 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 618921 T3035 oasc.ZkController.register We are http://127.0.0.1:33049/x_/rw/collection1/ and leader is http://127.0.0.1:33049/x_/rw/collection1/
[junit4:junit4] 2> 618921 T3035 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33049/x_/rw
[junit4:junit4] 2> 618921 T3035 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 618922 T3035 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 618922 T3035 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 618924 T3035 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 618924 T2992 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 618925 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 618925 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 618990 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 618992 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40786
[junit4:junit4] 2> 618993 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 618993 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 618993 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596
[junit4:junit4] 2> 618994 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596/solr.xml
[junit4:junit4] 2> 618994 T2992 oasc.CoreContainer.<init> New CoreContainer 5234384
[junit4:junit4] 2> 618994 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596/'
[junit4:junit4] 2> 618995 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596/'
[junit4:junit4] 2> 619045 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 619045 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 619046 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 619046 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 619046 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 619047 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 619047 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 619047 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 619047 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 619048 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 619063 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 619064 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4] 2> 619064 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 619065 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 619066 T3048 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ac279 name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 619067 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 619068 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 619071 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 619072 T3050 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@da84e1 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 619072 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 619075 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 620147 T2992 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40786_x_%2Frw
[junit4:junit4] 2> 620449 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40786_x_%2Frw
[junit4:junit4] 2> 620460 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 620464 T3014 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 620466 T3021 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 620469 T3034 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 620467 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 620471 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 620476 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 620480 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33049_x_%2Frw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33049/x_/rw"}
[junit4:junit4] 2> 620494 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 620494 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 620503 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 620710 T3050 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 620711 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 620930 T3051 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 620934 T3051 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 620937 T3051 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 622014 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 622026 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40786_x_%2Frw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40786/x_/rw"}
[junit4:junit4] 2> 622036 T3015 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 622036 T3015 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4] 2> 622049 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 622049 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 622051 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 622051 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 622939 T3051 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 622964 T3051 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596/collection1
[junit4:junit4] 2> 622964 T3051 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 622977 T3051 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 622977 T3051 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 622986 T3051 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596/collection1/'
[junit4:junit4] 2> 622999 T3051 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596/collection1/lib/README' to classloader
[junit4:junit4] 2> 623000 T3051 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 623714 T3051 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 624005 T3051 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 624012 T3051 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 624026 T3051 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 624892 T3051 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 624899 T3051 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 624902 T3051 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 624917 T3051 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 624921 T3051 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 624924 T3051 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 624926 T3051 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 624928 T3051 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 624928 T3051 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 624929 T3051 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 624929 T3051 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 624930 T3051 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 624931 T3051 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty2/
[junit4:junit4] 2> 624932 T3051 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff291e
[junit4:junit4] 2> 624935 T3051 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty2
[junit4:junit4] 2> 624936 T3051 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty2/index/
[junit4:junit4] 2> 624936 T3051 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 624936 T3051 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty2/index
[junit4:junit4] 2> 624958 T3051 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 624958 T3051 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 624962 T3051 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 624962 T3051 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 624963 T3051 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 624965 T3051 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 624966 T3051 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 624966 T3051 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 624967 T3051 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 624968 T3051 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 624968 T3051 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 624972 T3051 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 624978 T3051 oass.SolrIndexSearcher.<init> Opening Searcher@1ad25f9 main
[junit4:junit4] 2> 624980 T3051 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 624980 T3051 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 625053 T3053 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1ad25f9 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 625057 T3051 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 625057 T3051 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40786/x_/rw collection:collection1 shard:shard2
[junit4:junit4] 2> 625058 T3051 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 625064 T3051 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4:junit4] 2> 625067 T3051 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 625067 T3051 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 625067 T3051 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40786/x_/rw/collection1/
[junit4:junit4] 2> 625067 T3051 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 625067 T3051 oasc.SyncStrategy.syncToMe http://127.0.0.1:40786/x_/rw/collection1/ has no replicas
[junit4:junit4] 2> 625068 T3051 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40786/x_/rw/collection1/ shard2
[junit4:junit4] 2> 625068 T3051 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 625070 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 625076 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 625076 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 625076 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 625076 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 625123 T3051 oasc.ZkController.register We are http://127.0.0.1:40786/x_/rw/collection1/ and leader is http://127.0.0.1:40786/x_/rw/collection1/
[junit4:junit4] 2> 625123 T3051 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40786/x_/rw
[junit4:junit4] 2> 625123 T3051 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 625124 T3051 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 625124 T3051 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 625126 T3051 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 625129 T2992 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 625130 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 625131 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 625243 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 625260 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40386
[junit4:junit4] 2> 625261 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 625261 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 625262 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811
[junit4:junit4] 2> 625262 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811/solr.xml
[junit4:junit4] 2> 625262 T2992 oasc.CoreContainer.<init> New CoreContainer 686762
[junit4:junit4] 2> 625263 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811/'
[junit4:junit4] 2> 625263 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811/'
[junit4:junit4] 2> 625319 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 625320 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 625320 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 625320 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 625321 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 625321 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 625321 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 625322 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 625322 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 625323 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 625340 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 625341 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4] 2> 625342 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 625344 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 625349 T3065 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@121f2c7 name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 625349 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 625350 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 625356 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 625357 T3067 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@84839e name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 625357 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 625360 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 626364 T2992 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40386_x_%2Frw
[junit4:junit4] 2> 626365 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40386_x_%2Frw
[junit4:junit4] 2> 626366 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 626374 T3021 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 626374 T3067 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 626374 T3014 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 626374 T3050 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 626374 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 626375 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 626374 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 626375 T3034 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 626379 T3068 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 626379 T3068 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 626380 T3068 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 626584 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 626585 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40786_x_%2Frw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40786/x_/rw"}
[junit4:junit4] 2> 626588 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40386_x_%2Frw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40386/x_/rw"}
[junit4:junit4] 2> 626588 T3015 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 626588 T3015 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4:junit4] 2> 626591 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 626591 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 626591 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 626591 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 626591 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 627380 T3068 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 627381 T3068 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811/collection1
[junit4:junit4] 2> 627381 T3068 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 627382 T3068 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 627382 T3068 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 627384 T3068 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811/collection1/'
[junit4:junit4] 2> 627386 T3068 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811/collection1/lib/README' to classloader
[junit4:junit4] 2> 627386 T3068 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 627438 T3068 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 627477 T3068 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 627478 T3068 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 627483 T3068 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 627897 T3068 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 627903 T3068 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 627906 T3068 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 627918 T3068 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 627923 T3068 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 627928 T3068 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 627930 T3068 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 627930 T3068 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 627930 T3068 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 627931 T3068 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 627931 T3068 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 627932 T3068 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 627932 T3068 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty3/
[junit4:junit4] 2> 627932 T3068 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff291e
[junit4:junit4] 2> 627933 T3068 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty3
[junit4:junit4] 2> 627934 T3068 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty3/index/
[junit4:junit4] 2> 627934 T3068 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 627935 T3068 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty3/index
[junit4:junit4] 2> 627938 T3068 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 627939 T3068 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 627941 T3068 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 627941 T3068 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 627941 T3068 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 627942 T3068 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 627943 T3068 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 627943 T3068 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 627943 T3068 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 627944 T3068 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 627944 T3068 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 627947 T3068 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 627951 T3068 oass.SolrIndexSearcher.<init> Opening Searcher@1e6ce72 main
[junit4:junit4] 2> 627952 T3068 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 627952 T3068 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 627957 T3069 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e6ce72 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 627960 T3068 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 627960 T3068 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40386/x_/rw collection:collection1 shard:shard3
[junit4:junit4] 2> 627960 T3068 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 627964 T3068 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4:junit4] 2> 627966 T3068 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 627966 T3068 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 627966 T3068 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40386/x_/rw/collection1/
[junit4:junit4] 2> 627967 T3068 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 627967 T3068 oasc.SyncStrategy.syncToMe http://127.0.0.1:40386/x_/rw/collection1/ has no replicas
[junit4:junit4] 2> 627967 T3068 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40386/x_/rw/collection1/ shard3
[junit4:junit4] 2> 627968 T3068 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 628098 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 628102 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 628102 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 628102 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 628102 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 628102 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 628121 T3068 oasc.ZkController.register We are http://127.0.0.1:40386/x_/rw/collection1/ and leader is http://127.0.0.1:40386/x_/rw/collection1/
[junit4:junit4] 2> 628121 T3068 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40386/x_/rw
[junit4:junit4] 2> 628121 T3068 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 628122 T3068 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 628122 T3068 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 628124 T3068 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 628125 T2992 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 628125 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 628125 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 628203 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 628212 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42828
[junit4:junit4] 2> 628212 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 628213 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 628213 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797
[junit4:junit4] 2> 628213 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797/solr.xml
[junit4:junit4] 2> 628214 T2992 oasc.CoreContainer.<init> New CoreContainer 17887150
[junit4:junit4] 2> 628214 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797/'
[junit4:junit4] 2> 628214 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797/'
[junit4:junit4] 2> 628268 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 628269 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 628269 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 628270 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 628270 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 628270 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 628271 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 628271 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 628271 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 628272 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 628275 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 628275 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4] 2> 628275 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 628276 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 628278 T3081 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e0d861 name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 628279 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 628280 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 628297 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 628298 T3083 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f1254c name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 628298 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 628301 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 629303 T2992 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42828_x_%2Frw
[junit4:junit4] 2> 629305 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42828_x_%2Frw
[junit4:junit4] 2> 629306 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 629306 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 629312 T3021 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 629312 T3014 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 629312 T3050 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 629312 T3083 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 629312 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 629312 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 629312 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 629313 T3067 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 629313 T3034 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 629318 T3084 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 629319 T3084 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 629319 T3084 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 629608 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 629609 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40386_x_%2Frw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40386/x_/rw"}
[junit4:junit4] 2> 629613 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42828_x_%2Frw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42828/x_/rw"}
[junit4:junit4] 2> 629613 T3015 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 629613 T3015 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 629616 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 629616 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 629616 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 629616 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 629616 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 629616 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 630320 T3084 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 630320 T3084 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797/collection1
[junit4:junit4] 2> 630320 T3084 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 630321 T3084 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 630322 T3084 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 630323 T3084 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797/collection1/'
[junit4:junit4] 2> 630323 T3084 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797/collection1/lib/README' to classloader
[junit4:junit4] 2> 630324 T3084 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 630358 T3084 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 630395 T3084 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 630396 T3084 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 630400 T3084 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 630806 T3084 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 630812 T3084 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 630815 T3084 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 630824 T3084 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 630827 T3084 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 630830 T3084 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 630831 T3084 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 630831 T3084 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 630831 T3084 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 630832 T3084 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 630832 T3084 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 630832 T3084 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 630833 T3084 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty4/
[junit4:junit4] 2> 630833 T3084 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff291e
[junit4:junit4] 2> 630833 T3084 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty4
[junit4:junit4] 2> 630834 T3084 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty4/index/
[junit4:junit4] 2> 630834 T3084 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 630834 T3084 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty4/index
[junit4:junit4] 2> 630837 T3084 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 630837 T3084 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 630839 T3084 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 630839 T3084 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 630839 T3084 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 630840 T3084 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 630840 T3084 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 630841 T3084 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 630841 T3084 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 630841 T3084 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 630842 T3084 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 630844 T3084 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 630847 T3084 oass.SolrIndexSearcher.<init> Opening Searcher@c4a32c main
[junit4:junit4] 2> 630848 T3084 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 630848 T3084 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 630853 T3085 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c4a32c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 630855 T3084 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 630855 T3084 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42828/x_/rw collection:collection1 shard:shard1
[junit4:junit4] 2> 630858 T3084 oasc.ZkController.register We are http://127.0.0.1:42828/x_/rw/collection1/ and leader is http://127.0.0.1:33049/x_/rw/collection1/
[junit4:junit4] 2> 630858 T3084 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42828/x_/rw
[junit4:junit4] 2> 630859 T3084 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 630859 T3084 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C493 name=collection1 org.apache.solr.core.SolrCore@1feff0f url=http://127.0.0.1:42828/x_/rw/collection1 node=127.0.0.1:42828_x_%2Frw C493_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:42828_x_%2Frw, base_url=http://127.0.0.1:42828/x_/rw}
[junit4:junit4] 2> 630859 T3086 C493 P42828 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 630860 T3084 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 630864 T3086 C493 P42828 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 630865 T3086 C493 P42828 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 630865 T3086 C493 P42828 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 630866 T2992 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 630866 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 630866 T3086 C493 P42828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 630867 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 630877 T3028 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 630932 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 630936 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42858
[junit4:junit4] 2> 630936 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 630937 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 630937 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537
[junit4:junit4] 2> 630937 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537/solr.xml
[junit4:junit4] 2> 630938 T2992 oasc.CoreContainer.<init> New CoreContainer 3800004
[junit4:junit4] 2> 630938 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537/'
[junit4:junit4] 2> 630938 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537/'
[junit4:junit4] 2> 630989 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 630990 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 630990 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 630990 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 630991 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 630991 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 630991 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 630991 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 630992 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 630992 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 630995 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 630996 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4] 2> 630996 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 630997 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 630998 T3098 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cb10b0 name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 630999 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 631000 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 631004 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 631005 T3100 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1719f84 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 631005 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 631008 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 631122 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 631123 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42828_x_%2Frw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42828/x_/rw"}
[junit4:junit4] 2> 631126 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 631126 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 631126 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 631126 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 631126 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 631126 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 631126 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 631879 T3028 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 631880 T3028 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node4&state=recovering&nodeName=127.0.0.1:42828_x_%252Frw&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1003
[junit4:junit4] 2> 632011 T2992 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42858_x_%2Frw
[junit4:junit4] 2> 632012 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42858_x_%2Frw
[junit4:junit4] 2> 632014 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 632014 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 632015 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 632015 T3050 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 632015 T3021 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 632016 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 632015 T3014 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 632016 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 632016 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 632017 T3100 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 632016 T3067 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 632017 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 632018 T3083 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 632018 T3034 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 632042 T3101 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 632042 T3101 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 632043 T3101 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 632631 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 632632 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42858_x_%2Frw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42858/x_/rw"}
[junit4:junit4] 2> 632632 T3015 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 632632 T3015 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4] 2> 632637 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 632637 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 632637 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 632637 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 632637 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 632637 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 632637 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 633043 T3101 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 633044 T3101 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537/collection1
[junit4:junit4] 2> 633045 T3101 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 633046 T3101 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 633046 T3101 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 633048 T3101 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537/collection1/'
[junit4:junit4] 2> 633049 T3101 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537/collection1/lib/README' to classloader
[junit4:junit4] 2> 633050 T3101 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 633110 T3101 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 633152 T3101 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 633153 T3101 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 633157 T3101 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 633590 T3101 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 633596 T3101 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 633598 T3101 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 633608 T3101 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 633611 T3101 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 633614 T3101 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 633615 T3101 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 633615 T3101 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 633615 T3101 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 633616 T3101 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 633616 T3101 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 633616 T3101 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 633617 T3101 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty5/
[junit4:junit4] 2> 633617 T3101 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff291e
[junit4:junit4] 2> 633617 T3101 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty5
[junit4:junit4] 2> 633618 T3101 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty5/index/
[junit4:junit4] 2> 633618 T3101 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 633618 T3101 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty5/index
[junit4:junit4] 2> 633768 T3101 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 633768 T3101 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 633770 T3101 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 633770 T3101 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 633771 T3101 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 633771 T3101 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 633772 T3101 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 633772 T3101 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 633773 T3101 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 633773 T3101 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 633774 T3101 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 633776 T3101 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 633779 T3101 oass.SolrIndexSearcher.<init> Opening Searcher@19032b8 main
[junit4:junit4] 2> 633779 T3101 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 633780 T3101 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 633785 T3102 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19032b8 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 633788 T3101 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 633788 T3101 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42858/x_/rw collection:collection1 shard:shard2
[junit4:junit4] 2> 633791 T3101 oasc.ZkController.register We are http://127.0.0.1:42858/x_/rw/collection1/ and leader is http://127.0.0.1:40786/x_/rw/collection1/
[junit4:junit4] 2> 633791 T3101 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42858/x_/rw
[junit4:junit4] 2> 633791 T3101 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 633791 T3101 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> 633792 T3101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C494 name=collection1 org.apache.solr.core.SolrCore@1ffb77f url=http://127.0.0.1:42858/x_/rw/collection1 node=127.0.0.1:42858_x_%2Frw C494_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:42858_x_%2Frw, base_url=http://127.0.0.1:42858/x_/rw}
[junit4:junit4] 2> 633793 T3103 C494 P42858 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 633799 T3103 C494 P42858 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 633799 T2992 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 633799 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 633799 T3103 C494 P42858 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 633800 T3103 C494 P42858 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 633800 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 633801 T3103 C494 P42858 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 633805 T3044 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 633864 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 633877 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33030
[junit4:junit4] 2> 633878 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 633878 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 633878 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471
[junit4:junit4] 2> 633879 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471/solr.xml
[junit4:junit4] 2> 633879 T2992 oasc.CoreContainer.<init> New CoreContainer 25622568
[junit4:junit4] 2> 633879 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471/'
[junit4:junit4] 2> 633880 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471/'
[junit4:junit4] 2> ASYNC NEW_CORE C495 name=collection1 org.apache.solr.core.SolrCore@1feff0f url=http://127.0.0.1:42828/x_/rw/collection1 node=127.0.0.1:42828_x_%2Frw C495_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:42828_x_%2Frw, base_url=http://127.0.0.1:42828/x_/rw}
[junit4:junit4] 2> 633882 T3086 C495 P42828 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:33049/x_/rw/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 633883 T3086 C495 P42828 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:42828/x_/rw START replicas=[http://127.0.0.1:33049/x_/rw/collection1/] nUpdates=100
[junit4:junit4] 2> 633883 T3086 C495 P42828 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 633883 T3086 C495 P42828 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 633884 T3086 C495 P42828 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 633884 T3086 C495 P42828 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 633884 T3086 C495 P42828 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 633884 T3086 C495 P42828 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:33049/x_/rw/collection1/. core=collection1
[junit4:junit4] 2> 633884 T3086 C495 P42828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C496 name=collection1 org.apache.solr.core.SolrCore@2ebc10 url=http://127.0.0.1:33049/x_/rw/collection1 node=127.0.0.1:33049_x_%2Frw C496_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:33049_x_%2Frw, base_url=http://127.0.0.1:33049/x_/rw, leader=true}
[junit4:junit4] 2> 633896 T3030 C496 P33049 oasc.SolrCore.execute [collection1] webapp=/x_/rw path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=2
[junit4:junit4] 2> 633899 T3028 C496 P33049 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 633902 T3028 C496 P33049 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 633909 T3028 C496 P33049 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 633909 T3028 C496 P33049 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 633910 T3028 C496 P33049 oass.SolrIndexSearcher.<init> Opening Searcher@1a024f9 realtime
[junit4:junit4] 2> 633911 T3028 C496 P33049 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 633911 T3028 C496 P33049 oasup.LogUpdateProcessor.finish [collection1] webapp=/x_/rw path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 14
[junit4:junit4] 2> 633912 T3086 C495 P42828 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 633912 T3086 C495 P42828 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 633916 T3027 C496 P33049 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 633916 T3027 C496 P33049 oasc.SolrCore.execute [collection1] webapp=/x_/rw path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 633917 T3086 C495 P42828 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 633917 T3086 C495 P42828 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 633918 T3086 C495 P42828 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 633918 T3086 C495 P42828 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 633919 T3086 C495 P42828 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 633970 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 633971 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 633971 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 633971 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 633971 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 633972 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 633972 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 633973 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 633973 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 633974 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 633977 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 633977 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4] 2> 633978 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 633978 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 633980 T3116 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@143a012 name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 633981 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 633986 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 633991 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 633992 T3118 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6152e5 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 633993 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 633996 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 634142 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 634143 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42858_x_%2Frw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42858/x_/rw"}
[junit4:junit4] 2> 634146 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42828_x_%2Frw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42828/x_/rw"}
[junit4:junit4] 2> 634159 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 634160 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 634160 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 634160 T3118 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 634160 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 634160 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 634159 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 634159 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 634806 T3044 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 634806 T3044 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node5&state=recovering&nodeName=127.0.0.1:42858_x_%252Frw&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 635001 T2992 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33030_x_%2Frw
[junit4:junit4] 2> 635003 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33030_x_%2Frw
[junit4:junit4] 2> 635004 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 635005 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 635005 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 635005 T3118 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 635006 T3118 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 635006 T3100 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 635005 T3021 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 635007 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 635007 T3067 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 635007 T3050 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 635006 T3014 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 635008 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 635008 T3034 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 635007 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 635008 T3083 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 635008 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 635013 T3119 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 635013 T3119 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 635014 T3119 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 635666 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 635667 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33030_x_%2Frw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33030/x_/rw"}
[junit4:junit4] 2> 635667 T3015 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 635667 T3015 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4:junit4] 2> 635716 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 635716 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 635716 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 635716 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 635716 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 635716 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 635716 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 635716 T3118 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 636014 T3119 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 636015 T3119 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471/collection1
[junit4:junit4] 2> 636015 T3119 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 636016 T3119 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 636016 T3119 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 636017 T3119 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471/collection1/'
[junit4:junit4] 2> 636018 T3119 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471/collection1/lib/README' to classloader
[junit4:junit4] 2> 636018 T3119 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 636056 T3119 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 636096 T3119 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 636097 T3119 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 636101 T3119 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 636552 T3119 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 636558 T3119 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 636561 T3119 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 636571 T3119 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 636574 T3119 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 636577 T3119 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 636578 T3119 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 636578 T3119 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 636578 T3119 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 636579 T3119 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 636579 T3119 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 636579 T3119 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 636579 T3119 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty6/
[junit4:junit4] 2> 636580 T3119 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff291e
[junit4:junit4] 2> 636580 T3119 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty6
[junit4:junit4] 2> 636581 T3119 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty6/index/
[junit4:junit4] 2> 636581 T3119 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 636581 T3119 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty6/index
[junit4:junit4] 2> 636593 T3119 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 636593 T3119 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 636595 T3119 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 636595 T3119 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 636596 T3119 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 636596 T3119 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 636597 T3119 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 636597 T3119 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 636597 T3119 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 636598 T3119 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 636598 T3119 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 636601 T3119 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 636605 T3119 oass.SolrIndexSearcher.<init> Opening Searcher@727a6 main
[junit4:junit4] 2> 636606 T3119 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 636606 T3119 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 636612 T3120 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@727a6 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 636614 T3119 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 636614 T3119 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33030/x_/rw collection:collection1 shard:shard3
[junit4:junit4] 2> 636618 T3119 oasc.ZkController.register We are http://127.0.0.1:33030/x_/rw/collection1/ and leader is http://127.0.0.1:40386/x_/rw/collection1/
[junit4:junit4] 2> 636618 T3119 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33030/x_/rw
[junit4:junit4] 2> 636618 T3119 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 636618 T3119 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C497 name=collection1 org.apache.solr.core.SolrCore@11e2f19 url=http://127.0.0.1:33030/x_/rw/collection1 node=127.0.0.1:33030_x_%2Frw C497_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:33030_x_%2Frw, base_url=http://127.0.0.1:33030/x_/rw}
[junit4:junit4] 2> 636619 T3121 C497 P33030 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 636622 T3121 C497 P33030 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 636622 T3119 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 636622 T3121 C497 P33030 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 636623 T3121 C497 P33030 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 636623 T2992 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 636624 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 636624 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 636626 T3121 C497 P33030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 636630 T3063 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node6, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 636691 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 636694 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47110
[junit4:junit4] 2> 636695 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 636695 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 636695 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295
[junit4:junit4] 2> 636696 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295/solr.xml
[junit4:junit4] 2> 636696 T2992 oasc.CoreContainer.<init> New CoreContainer 13795504
[junit4:junit4] 2> 636697 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295/'
[junit4:junit4] 2> 636697 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295/'
[junit4:junit4] 2> 636788 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 636789 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 636789 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 636790 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 636790 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 636791 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 636791 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 636792 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 636792 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 636792 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 636797 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 636797 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4] 2> 636798 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 636799 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 636800 T3133 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a8280b name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 636802 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 636804 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> ASYNC NEW_CORE C498 name=collection1 org.apache.solr.core.SolrCore@1ffb77f url=http://127.0.0.1:42858/x_/rw/collection1 node=127.0.0.1:42858_x_%2Frw C498_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:42858_x_%2Frw, base_url=http://127.0.0.1:42858/x_/rw}
[junit4:junit4] 2> 636807 T3103 C498 P42858 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:40786/x_/rw/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 636807 T3103 C498 P42858 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:42858/x_/rw START replicas=[http://127.0.0.1:40786/x_/rw/collection1/] nUpdates=100
[junit4:junit4] 2> 636808 T3103 C498 P42858 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 636808 T3103 C498 P42858 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 636808 T3103 C498 P42858 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 636808 T3103 C498 P42858 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 636809 T3103 C498 P42858 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 636809 T3103 C498 P42858 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:40786/x_/rw/collection1/. core=collection1
[junit4:junit4] 2> 636809 T3103 C498 P42858 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 636814 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 636816 T3136 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ea8ed1 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 636816 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 636820 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C499 name=collection1 org.apache.solr.core.SolrCore@92cef2 url=http://127.0.0.1:40786/x_/rw/collection1 node=127.0.0.1:40786_x_%2Frw C499_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:40786_x_%2Frw, base_url=http://127.0.0.1:40786/x_/rw, leader=true}
[junit4:junit4] 2> 636821 T3044 C499 P40786 oasc.SolrCore.execute [collection1] webapp=/x_/rw path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=2
[junit4:junit4] 2> 636824 T3043 C499 P40786 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 636827 T3043 C499 P40786 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 636827 T3043 C499 P40786 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 636828 T3043 C499 P40786 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 636828 T3043 C499 P40786 oass.SolrIndexSearcher.<init> Opening Searcher@1adc008 realtime
[junit4:junit4] 2> 636829 T3043 C499 P40786 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 636829 T3043 C499 P40786 oasup.LogUpdateProcessor.finish [collection1] webapp=/x_/rw path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 636830 T3103 C498 P42858 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 636830 T3103 C498 P42858 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 636832 T3045 C499 P40786 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 636833 T3045 C499 P40786 oasc.SolrCore.execute [collection1] webapp=/x_/rw path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 636833 T3103 C498 P42858 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 636833 T3103 C498 P42858 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 636834 T3103 C498 P42858 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 636834 T3103 C498 P42858 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 636835 T3103 C498 P42858 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 637222 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 637222 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33030_x_%2Frw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33030/x_/rw"}
[junit4:junit4] 2> 637225 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42858_x_%2Frw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42858/x_/rw"}
[junit4:junit4] 2> 637229 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 637230 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 637230 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 637230 T3136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 637230 T3118 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 637230 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 637230 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 637230 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 637230 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 637631 T3063 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node6, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 637631 T3063 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node6&state=recovering&nodeName=127.0.0.1:33030_x_%252Frw&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 637824 T2992 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47110_x_%2Frw
[junit4:junit4] 2> 637825 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47110_x_%2Frw
[junit4:junit4] 2> 637826 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 637827 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 637827 T3021 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 637827 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 637826 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 637827 T3118 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 637827 T3100 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 637828 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 637828 T3050 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 637829 T3034 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 637828 T3136 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 637828 T3118 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 637830 T3014 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 637830 T3136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 637829 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 637830 T3067 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 637830 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 637831 T3083 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 637839 T3137 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 637839 T3137 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 637840 T3137 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 638739 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 638740 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node7",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:47110_x_%2Frw",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47110/x_/rw"}
[junit4:junit4] 2> 638740 T3015 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 638740 T3015 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 638796 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 638796 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 638796 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 638796 T3136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 638796 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 638796 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 638796 T3118 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 638796 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 638796 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 638840 T3137 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 638841 T3137 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295/collection1
[junit4:junit4] 2> 638841 T3137 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 638842 T3137 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 638842 T3137 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 638843 T3137 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295/collection1/'
[junit4:junit4] 2> 638844 T3137 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295/collection1/lib/README' to classloader
[junit4:junit4] 2> 638844 T3137 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 638881 T3137 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 638929 T3137 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 638930 T3137 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 638934 T3137 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 639364 T3137 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 639370 T3137 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 639372 T3137 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 639383 T3137 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 639386 T3137 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 639389 T3137 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 639390 T3137 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 639390 T3137 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 639390 T3137 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 639391 T3137 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 639391 T3137 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 639392 T3137 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 639392 T3137 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty7/
[junit4:junit4] 2> 639392 T3137 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff291e
[junit4:junit4] 2> 639393 T3137 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty7
[junit4:junit4] 2> 639393 T3137 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty7/index/
[junit4:junit4] 2> 639393 T3137 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty7/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 639394 T3137 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty7/index
[junit4:junit4] 2> 639397 T3137 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty7/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 639397 T3137 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 639399 T3137 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 639399 T3137 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 639400 T3137 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 639400 T3137 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 639401 T3137 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 639401 T3137 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 639402 T3137 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 639402 T3137 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 639403 T3137 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 639405 T3137 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 639408 T3137 oass.SolrIndexSearcher.<init> Opening Searcher@18a2663 main
[junit4:junit4] 2> 639409 T3137 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 639409 T3137 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 639416 T3138 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@18a2663 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 639419 T3137 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 639419 T3137 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47110/x_/rw collection:collection1 shard:shard1
[junit4:junit4] 2> 639421 T3137 oasc.ZkController.register We are http://127.0.0.1:47110/x_/rw/collection1/ and leader is http://127.0.0.1:33049/x_/rw/collection1/
[junit4:junit4] 2> 639421 T3137 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47110/x_/rw
[junit4:junit4] 2> 639421 T3137 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 639421 T3137 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C500 name=collection1 org.apache.solr.core.SolrCore@6d95cf url=http://127.0.0.1:47110/x_/rw/collection1 node=127.0.0.1:47110_x_%2Frw C500_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:47110_x_%2Frw, base_url=http://127.0.0.1:47110/x_/rw}
[junit4:junit4] 2> 639422 T3139 C500 P47110 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 639422 T3137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 639425 T3139 C500 P47110 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 639425 T3139 C500 P47110 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 639426 T3139 C500 P47110 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 639426 T2992 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 639426 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 639427 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 639429 T3139 C500 P47110 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 639434 T3030 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node7, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 639499 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 639501 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33646
[junit4:junit4] 2> 639502 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 639502 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 639502 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1372413463098
[junit4:junit4] 2> 639503 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1372413463098/solr.xml
[junit4:junit4] 2> 639503 T2992 oasc.CoreContainer.<init> New CoreContainer 2223439
[junit4:junit4] 2> 639503 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1372413463098/'
[junit4:junit4] 2> 639504 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1372413463098/'
[junit4:junit4] 2> 639578 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 639578 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 639579 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 639579 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 639579 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 639580 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 639580 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 639580 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 639581 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 639581 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 639584 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 639585 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4] 2> 639585 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 639586 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 639587 T3151 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@94c5a4 name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 639588 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 639589 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 639593 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 639594 T3153 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1375e23 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 639594 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 639596 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C501 name=collection1 org.apache.solr.core.SolrCore@11e2f19 url=http://127.0.0.1:33030/x_/rw/collection1 node=127.0.0.1:33030_x_%2Frw C501_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:33030_x_%2Frw, base_url=http://127.0.0.1:33030/x_/rw}
[junit4:junit4] 2> 639632 T3121 C501 P33030 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:40386/x_/rw/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 639633 T3121 C501 P33030 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:33030/x_/rw START replicas=[http://127.0.0.1:40386/x_/rw/collection1/] nUpdates=100
[junit4:junit4] 2> 639633 T3121 C501 P33030 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 639634 T3121 C501 P33030 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 639634 T3121 C501 P33030 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 639634 T3121 C501 P33030 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 639634 T3121 C501 P33030 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 639635 T3121 C501 P33030 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:40386/x_/rw/collection1/. core=collection1
[junit4:junit4] 2> 639635 T3121 C501 P33030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C502 name=collection1 org.apache.solr.core.SolrCore@e58d0c url=http://127.0.0.1:40386/x_/rw/collection1 node=127.0.0.1:40386_x_%2Frw C502_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:40386_x_%2Frw, base_url=http://127.0.0.1:40386/x_/rw, leader=true}
[junit4:junit4] 2> 639639 T3062 C502 P40386 oasc.SolrCore.execute [collection1] webapp=/x_/rw path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 639639 T3060 C502 P40386 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 639641 T3060 C502 P40386 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 639642 T3060 C502 P40386 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 639642 T3060 C502 P40386 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit
[...truncated too long message...]
39 T2992 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 45695
[junit4:junit4] 2> 691377 T2992 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 691378 T2992 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51741 51741
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> 691382 T2992 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 79147 T2991 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 4 leaked thread(s).
[junit4:junit4] 2> 691458 T3207 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 691458 T3207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 691459 T3207 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 99287 T2991 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 1 thread leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:
[junit4:junit4] 2> 1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
[junit4:junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] 2> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
[junit4:junit4] 2> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4:junit4] 2> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
[junit4:junit4] 2> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 99287 T2991 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4:junit4] 2> 1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
[junit4:junit4] 2> 102289 T2991 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4:junit4] 2> 1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
[junit4:junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] 2> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
[junit4:junit4] 2> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4:junit4] 2> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
[junit4:junit4] 2> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=PostingsFormat(name=MockRandom), text=PostingsFormat(name=Direct), _version_=PostingsFormat(name=MockRandom), rnd_b=Pulsing41(freqCutoff=5 minBlockSize=53 maxBlockSize=114), intDefault=PostingsFormat(name=MockRandom), timestamp=PostingsFormat(name=MockRandom), id=Pulsing41(freqCutoff=5 minBlockSize=53 maxBlockSize=114), a_t=PostingsFormat(name=MockRandom), range_facet_sl=PostingsFormat(name=Direct), range_facet_si=PostingsFormat(name=NestedPulsing), other_tl1=PostingsFormat(name=MockRandom), multiDefault=PostingsFormat(name=NestedPulsing), a_si=PostingsFormat(name=NestedPulsing)}, docValues:{timestamp=DocValuesFormat(name=CheapBastard)}, sim=DefaultSimilarity, locale=es_PA, timezone=America/Chicago
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation 1.7.0_25 (32-bit)/cpus=8,threads=1,free=167126544,total=383561728
[junit4:junit4] 2> NOTE: All tests run in this JVM: [OpenCloseCoreStressTest, RequestHandlersTest, TestReload, TestJmxMonitoredMap, TestRecovery, FullSolrCloudDistribCmdsTest, TestPropInject, SampleTest, FieldAnalysisRequestHandlerTest, SpellCheckComponentTest, TestIndexingPerformance, TestWordDelimiterFilterFactory, LeaderElectionIntegrationTest, TestFoldingMultitermQuery, TestUniqueKeyFieldResource, TestSchemaResource, TestJmxIntegration, TestRecoveryHdfs, TermVectorComponentDistributedTest, DOMUtilTest, TestAnalyzedSuggestions, EchoParamsTest, TestCloudManagedSchema, HighlighterTest, TestRTGBase, BasicDistributedZk2Test, TestStressLucene, SpellPossibilityIteratorTest, ZkNodePropsTest, TestLMJelinekMercerSimilarityFactory, ShardRoutingCustomTest, TestUpdate, SimplePostToolTest, TestDefaultSimilarityFactory, TestSolrJ, DebugComponentTest, CachingDirectoryFactoryTest, TestMultiCoreConfBootstrap, ConvertedLegacyTest, PingRequestHandlerTest, ShardSplitTest, UpdateParamsTest, UniqFieldsUpdateProcessorFactoryTest, TestIndexSearcher, TestSolrXMLSerializer, TestBinaryField, HdfsBasicDistributedZk2Test, TestCoreDiscovery, ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=7BB20C2B3A27495C -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_PA -Dtests.timezone=America/Chicago -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
[junit4:junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:
[junit4:junit4] > 1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
[junit4:junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
[junit4:junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4:junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
[junit4:junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([7BB20C2B3A27495C]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4] > 1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
[junit4:junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
[junit4:junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4:junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
[junit4:junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([7BB20C2B3A27495C]:0)
[junit4:junit4] Completed on J1 in 102.37s, 1 test, 2 errors <<< FAILURES!
[...truncated 563 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:190: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:443: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1245: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:889: There were test failures: 312 suites, 1316 tests, 2 suite-level errors, 694 ignored (6 assumptions)
Total time: 31 minutes 23 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_25 -client -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_25) - Build #
6347 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6347/
Java: 64bit/jdk1.7.0_25 -XX:+UseCompressedOops -XX:+UseSerialGC
1 tests failed.
FAILED: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:43365/collection1lastClient and got 256 from http://127.0.0.1:55874/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:43365/collection1lastClient and got 256 from http://127.0.0.1:55874/collection1
at __randomizedtesting.SeedInfo.seed([EE579D43B894FBE1:6FB1135BCFCB9BDD]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1045)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:237)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9219 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4] 2> 31314 T3 oasu.ConcurrentLRUCache.finalize ERROR ConcurrentLRUCache was not destroyed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
[junit4:junit4] 2> 31317 T161 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 31323 T161 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1372408195692
[junit4:junit4] 2> 31326 T161 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 31329 T162 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 31429 T161 oasc.ZkTestServer.run start zk server on port:51822
[junit4:junit4] 2> 34479 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 34512 T168 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@664c6b6c name:ZooKeeperConnection Watcher:127.0.0.1:51822 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 34513 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 34514 T161 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 34538 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 34540 T170 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5bc1d381 name:ZooKeeperConnection Watcher:127.0.0.1:51822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 34540 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 34542 T161 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 34546 T161 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 34550 T161 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 34552 T161 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 34556 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 34556 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 34562 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 34562 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 34567 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 34567 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 34571 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 34572 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 34575 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 34576 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 34580 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 34580 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 34584 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 34584 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 34588 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 34589 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 34593 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 34593 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 34667 T161 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 34671 T161 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60204
[junit4:junit4] 2> 34672 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 34672 T161 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 34673 T161 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967
[junit4:junit4] 2> 34673 T161 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967/solr.xml
[junit4:junit4] 2> 34673 T161 oasc.CoreContainer.<init> New CoreContainer 720319641
[junit4:junit4] 2> 34674 T161 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967/'
[junit4:junit4] 2> 34674 T161 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967/'
[junit4:junit4] 2> 34725 T161 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 34725 T161 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 34726 T161 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 34726 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 34726 T161 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 34727 T161 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 34727 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 34727 T161 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 34727 T161 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 34728 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 34732 T161 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 34732 T161 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51822/solr
[junit4:junit4] 2> 34740 T161 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 34741 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 34744 T181 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2fcc2471 name:ZooKeeperConnection Watcher:127.0.0.1:51822 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 34744 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 34748 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 34753 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 34755 T183 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e8d95d name:ZooKeeperConnection Watcher:127.0.0.1:51822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 34755 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 34760 T161 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 34767 T161 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 34775 T161 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 34778 T161 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60204_
[junit4:junit4] 2> 34781 T161 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60204_
[junit4:junit4] 2> 34785 T161 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 34795 T161 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 34799 T161 oasc.Overseer.start Overseer (id=89942143517917187-127.0.0.1:60204_-n_0000000000) starting
[junit4:junit4] 2> 34807 T161 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 34817 T185 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 34817 T161 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 34820 T161 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 34823 T161 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 34830 T184 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 34834 T186 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 34834 T186 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 34836 T186 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 36337 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 36339 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:60204_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60204"}
[junit4:junit4] 2> 36339 T184 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 36341 T184 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 36351 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 36837 T186 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 36837 T186 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967/collection1
[junit4:junit4] 2> 36837 T186 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 36839 T186 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 36839 T186 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 36841 T186 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967/collection1/'
[junit4:junit4] 2> 36842 T186 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967/collection1/lib/README' to classloader
[junit4:junit4] 2> 36843 T186 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 36886 T186 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 36917 T186 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 36919 T186 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 36925 T186 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 37310 T186 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 37317 T186 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 37319 T186 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 37341 T186 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 37345 T186 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 37350 T186 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 37352 T186 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 37352 T186 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 37352 T186 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 37354 T186 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 37354 T186 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 37354 T186 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 37355 T186 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372408195692/control/data/
[junit4:junit4] 2> 37355 T186 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@522103ff
[junit4:junit4] 2> 37356 T186 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/control/data
[junit4:junit4] 2> 37356 T186 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372408195692/control/data/index/
[junit4:junit4] 2> 37356 T186 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372408195692/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 37356 T186 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/control/data/index
[junit4:junit4] 2> 37358 T186 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5324396a lockFactory=org.apache.lucene.store.NativeFSLockFactory@59e05bf9),segFN=segments_1,generation=1}
[junit4:junit4] 2> 37358 T186 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 37360 T186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 37361 T186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 37362 T186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 37362 T186 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 37363 T186 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 37363 T186 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 37363 T186 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 37364 T186 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 37364 T186 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 37366 T186 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 37369 T186 oass.SolrIndexSearcher.<init> Opening Searcher@62360ab1 main
[junit4:junit4] 2> 37370 T186 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 37370 T186 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 40380 T187 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@62360ab1 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 40383 T186 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 40384 T186 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60204 collection:control_collection shard:shard1
[junit4:junit4] 2> 40390 T186 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 40396 T186 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 40410 T186 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4] 2> 40416 T186 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 40417 T186 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 40419 T186 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60204/collection1/
[junit4:junit4] 2> 40420 T186 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 40420 T186 oasc.SyncStrategy.syncToMe http://127.0.0.1:60204/collection1/ has no replicas
[junit4:junit4] 2> 40421 T186 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60204/collection1/ shard1
[junit4:junit4] 2> 40421 T186 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 40863 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 40876 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 40890 T186 oasc.ZkController.register We are http://127.0.0.1:60204/collection1/ and leader is http://127.0.0.1:60204/collection1/
[junit4:junit4] 2> 40890 T186 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60204
[junit4:junit4] 2> 40890 T186 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 40890 T186 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 40891 T186 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 40893 T186 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 40894 T161 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 40895 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 40895 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 40899 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 40902 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 40903 T190 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42408073 name:ZooKeeperConnection Watcher:127.0.0.1:51822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 40904 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 40906 T161 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 40911 T161 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 40991 T161 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 40993 T161 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43365
[junit4:junit4] 2> 40994 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 40994 T161 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 40995 T161 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280
[junit4:junit4] 2> 40995 T161 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280/solr.xml
[junit4:junit4] 2> 40995 T161 oasc.CoreContainer.<init> New CoreContainer 1430281418
[junit4:junit4] 2> 40996 T161 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280/'
[junit4:junit4] 2> 40996 T161 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280/'
[junit4:junit4] 2> 41046 T161 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 41046 T161 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 41047 T161 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 41047 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 41047 T161 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 41047 T161 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 41048 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 41048 T161 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 41049 T161 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 41049 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 41053 T161 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 41054 T161 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51822/solr
[junit4:junit4] 2> 41054 T161 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 41055 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 41057 T201 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e5acdf name:ZooKeeperConnection Watcher:127.0.0.1:51822 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 41057 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 41063 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 41068 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 41070 T203 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@94c91f3 name:ZooKeeperConnection Watcher:127.0.0.1:51822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 41070 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 41075 T161 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 42080 T161 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43365_
[junit4:junit4] 2> 42081 T161 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43365_
[junit4:junit4] 2> 42085 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 42086 T190 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 42086 T203 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 42088 T183 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 42095 T204 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 42095 T204 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 42096 T204 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 42383 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 42385 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:60204_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60204"}
[junit4:junit4] 2> 42390 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:43365_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43365"}
[junit4:junit4] 2> 42390 T184 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 42390 T184 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 42397 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 42397 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 42397 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 43097 T204 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 43097 T204 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280/collection1
[junit4:junit4] 2> 43098 T204 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 43098 T204 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 43099 T204 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 43100 T204 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280/collection1/'
[junit4:junit4] 2> 43101 T204 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280/collection1/lib/README' to classloader
[junit4:junit4] 2> 43101 T204 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 43133 T204 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 43164 T204 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 43166 T204 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 43174 T204 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 43550 T204 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 43557 T204 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 43559 T204 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 43585 T204 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 43590 T204 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 43600 T204 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 43602 T204 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 43603 T204 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 43603 T204 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 43605 T204 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 43605 T204 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 43606 T204 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 43606 T204 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1/
[junit4:junit4] 2> 43607 T204 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@522103ff
[junit4:junit4] 2> 43608 T204 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1
[junit4:junit4] 2> 43609 T204 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1/index/
[junit4:junit4] 2> 43609 T204 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 43609 T204 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1/index
[junit4:junit4] 2> 43612 T204 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f5fb626 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7fd4880e),segFN=segments_1,generation=1}
[junit4:junit4] 2> 43612 T204 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 43615 T204 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 43616 T204 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 43617 T204 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 43618 T204 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 43619 T204 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 43619 T204 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 43620 T204 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 43621 T204 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 43622 T204 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 43626 T204 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 43629 T204 oass.SolrIndexSearcher.<init> Opening Searcher@2c81a8a main
[junit4:junit4] 2> 43631 T204 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 43631 T204 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 43636 T205 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2c81a8a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 43638 T204 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 43639 T204 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43365 collection:collection1 shard:shard1
[junit4:junit4] 2> 43640 T204 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 43648 T204 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4] 2> 43651 T204 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 43651 T204 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 43651 T204 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43365/collection1/
[junit4:junit4] 2> 43651 T204 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 43651 T204 oasc.SyncStrategy.syncToMe http://127.0.0.1:43365/collection1/ has no replicas
[junit4:junit4] 2> 43652 T204 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43365/collection1/ shard1
[junit4:junit4] 2> 43652 T204 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 43905 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 43921 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 43921 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 43921 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 43959 T204 oasc.ZkController.register We are http://127.0.0.1:43365/collection1/ and leader is http://127.0.0.1:43365/collection1/
[junit4:junit4] 2> 43960 T204 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43365
[junit4:junit4] 2> 43960 T204 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 43961 T204 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 43961 T204 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 43965 T204 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 43968 T161 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 43968 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 43969 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 44055 T161 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 44057 T161 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58508
[junit4:junit4] 2> 44058 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 44058 T161 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 44059 T161 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347
[junit4:junit4] 2> 44059 T161 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347/solr.xml
[junit4:junit4] 2> 44059 T161 oasc.CoreContainer.<init> New CoreContainer 408463892
[junit4:junit4] 2> 44060 T161 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347/'
[junit4:junit4] 2> 44060 T161 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347/'
[junit4:junit4] 2> 44114 T161 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 44115 T161 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 44115 T161 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 44116 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 44116 T161 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 44117 T161 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 44117 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 44118 T161 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 44118 T161 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 44119 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 44125 T161 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 44125 T161 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51822/solr
[junit4:junit4] 2> 44126 T161 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 44127 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 44130 T217 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30783ead name:ZooKeeperConnection Watcher:127.0.0.1:51822 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 44131 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 44134 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 44141 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 44144 T219 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2bdd746b name:ZooKeeperConnection Watcher:127.0.0.1:51822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 44145 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 44155 T161 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 45163 T161 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58508_
[junit4:junit4] 2> 45164 T161 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58508_
[junit4:junit4] 2> 45167 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 45167 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 45167 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 45169 T219 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 45171 T190 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 45171 T183 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 45171 T203 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 45179 T220 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 45180 T220 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 45181 T220 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 45433 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 45434 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:43365_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43365"}
[junit4:junit4] 2> 45439 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:58508_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:58508"}
[junit4:junit4] 2> 45439 T184 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 45440 T184 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 45445 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 45445 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 45445 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 45446 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 46181 T220 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 46182 T220 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347/collection1
[junit4:junit4] 2> 46183 T220 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 46184 T220 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 46185 T220 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 46188 T220 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347/collection1/'
[junit4:junit4] 2> 46190 T220 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347/collection1/lib/README' to classloader
[junit4:junit4] 2> 46191 T220 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 46238 T220 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 46275 T220 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 46277 T220 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 46281 T220 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 46660 T220 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 46668 T220 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 46671 T220 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 46696 T220 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 46701 T220 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 46705 T220 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 46707 T220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 46707 T220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 46708 T220 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 46709 T220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 46709 T220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 46710 T220 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 46710 T220 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty2/
[junit4:junit4] 2> 46710 T220 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@522103ff
[junit4:junit4] 2> 46711 T220 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty2
[junit4:junit4] 2> 46711 T220 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty2/index/
[junit4:junit4] 2> 46711 T220 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 46712 T220 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty2/index
[junit4:junit4] 2> 46714 T220 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1b88581a lockFactory=org.apache.lucene.store.NativeFSLockFactory@4127ea39),segFN=segments_1,generation=1}
[junit4:junit4] 2> 46714 T220 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 46717 T220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 46717 T220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 46718 T220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 46718 T220 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 46719 T220 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 46719 T220 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 46720 T220 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 46720 T220 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 46721 T220 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 46724 T220 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 46727 T220 oass.SolrIndexSearcher.<init> Opening Searcher@7b52d3fe main
[junit4:junit4] 2> 46727 T220 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 46728 T220 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 46736 T221 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7b52d3fe main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 46740 T220 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 46741 T220 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:58508 collection:collection1 shard:shard1
[junit4:junit4] 2> 46749 T220 oasc.ZkController.register We are http://127.0.0.1:58508/collection1/ and leader is http://127.0.0.1:43365/collection1/
[junit4:junit4] 2> 46749 T220 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:58508
[junit4:junit4] 2> 46749 T220 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 46750 T220 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C23 name=collection1 org.apache.solr.core.SolrCore@3266f7c5 url=http://127.0.0.1:58508/collection1 node=127.0.0.1:58508_ C23_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:58508_, base_url=http://127.0.0.1:58508}
[junit4:junit4] 2> 46754 T222 C23 P58508 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 46755 T222 C23 P58508 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 46755 T220 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 46756 T222 C23 P58508 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 46757 T222 C23 P58508 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 46758 T161 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 46759 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 46760 T222 C23 P58508 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 46760 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 46778 T197 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 46854 T161 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 46857 T161 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55874
[junit4:junit4] 2> 46857 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 46858 T161 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 46858 T161 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138
[junit4:junit4] 2> 46858 T161 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138/solr.xml
[junit4:junit4] 2> 46859 T161 oasc.CoreContainer.<init> New CoreContainer 383111009
[junit4:junit4] 2> 46859 T161 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138/'
[junit4:junit4] 2> 46859 T161 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138/'
[junit4:junit4] 2> 46913 T161 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 46914 T161 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 46914 T161 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 46915 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 46915 T161 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 46915 T161 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 46916 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 46916 T161 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 46917 T161 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 46917 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 46922 T161 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 46923 T161 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51822/solr
[junit4:junit4] 2> 46923 T161 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 46924 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 46926 T234 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a41c343 name:ZooKeeperConnection Watcher:127.0.0.1:51822 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 46927 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 46929 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 46935 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 46936 T236 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5cc01173 name:ZooKeeperConnection Watcher:127.0.0.1:51822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 46937 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 46942 T161 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 46960 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 46963 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:58508_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:58508"}
[junit4:junit4] 2> 46975 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 46976 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 46976 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 46976 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 46976 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 47779 T197 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 47780 T197 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node2&state=recovering&nodeName=127.0.0.1:58508_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1005
[junit4:junit4] 2> 47946 T161 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55874_
[junit4:junit4] 2> 47949 T161 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55874_
[junit4:junit4] 2> 47954 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 47954 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 47954 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 47954 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 47956 T236 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 47957 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 47961 T190 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 47963 T183 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 47963 T203 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 47963 T219 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 47971 T237 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 47971 T237 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 47974 T237 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 48488 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 48489 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55874_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55874"}
[junit4:junit4] 2> 48489 T184 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 48489 T184 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 48494 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 48494 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 48494 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 48495 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 48495 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 48975 T237 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 48976 T237 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138/collection1
[junit4:junit4] 2> 48976 T237 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 48978 T237 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 48978 T237 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 48982 T237 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138/collection1/'
[junit4:junit4] 2> 48983 T237 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138/collection1/lib/README' to classloader
[junit4:junit4] 2> 48983 T237 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 49033 T237 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 49081 T237 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 49084 T237 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 49088 T237 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 49435 T237 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 49441 T237 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 49444 T237 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 49465 T237 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 49468 T237 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 49472 T237 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 49473 T237 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 49474 T237 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 49474 T237 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 49475 T237 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 49476 T237 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 49476 T237 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 49476 T237 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty3/
[junit4:junit4] 2> 49476 T237 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@522103ff
[junit4:junit4] 2> 49477 T237 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty3
[junit4:junit4] 2> 49477 T237 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty3/index/
[junit4:junit4] 2> 49478 T237 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 49478 T237 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty3/index
[junit4:junit4] 2> 49479 T237 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4afd68fe lockFactory=org.apache.lucene.store.NativeFSLockFactory@5994519c),segFN=segments_1,generation=1}
[junit4:junit4] 2> 49480 T237 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 49482 T237 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 49482 T237 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 49483 T237 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 49483 T237 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 49484 T237 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 49484 T237 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 49485 T237 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 49485 T237 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 49486 T237 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 49488 T237 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 49490 T237 oass.SolrIndexSearcher.<init> Opening Searcher@ffe13fe main
[junit4:junit4] 2> 49491 T237 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 49491 T237 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 49496 T238 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ffe13fe main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 49498 T237 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 49498 T237 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55874 collection:collection1 shard:shard1
[junit4:junit4] 2> 49503 T237 oasc.ZkController.register We are http://127.0.0.1:55874/collection1/ and leader is http://127.0.0.1:43365/collection1/
[junit4:junit4] 2> 49503 T237 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55874
[junit4:junit4] 2> 49503 T237 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 49504 T237 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C24 name=collection1 org.apache.solr.core.SolrCore@151671f7 url=http://127.0.0.1:55874/collection1 node=127.0.0.1:55874_ C24_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:55874_, base_url=http://127.0.0.1:55874}
[junit4:junit4] 2> 49504 T239 C24 P55874 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 49505 T239 C24 P55874 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 49505 T237 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 49505 T239 C24 P55874 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 49506 T239 C24 P55874 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 49507 T161 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 49507 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 49508 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 49508 T239 C24 P55874 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 49513 T196 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 49578 T161 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 49581 T161 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57393
[junit4:junit4] 2> 49582 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 49583 T161 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 49583 T161 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881
[junit4:junit4] 2> 49583 T161 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881/solr.xml
[junit4:junit4] 2> 49583 T161 oasc.CoreContainer.<init> New CoreContainer 1726007250
[junit4:junit4] 2> 49584 T161 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881/'
[junit4:junit4] 2> 49584 T161 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881/'
[junit4:junit4] 2> 49634 T161 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 49634 T161 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 49634 T161 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 49635 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 49635 T161 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 49635 T161 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 49636 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 49636 T161 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 49636 T161 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 49637 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 49640 T161 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 49641 T161 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51822/solr
[junit4:junit4] 2> 49641 T161 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 49642 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 49644 T251 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d2703fb name:ZooKeeperConnection Watcher:127.0.0.1:51822 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 49645 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 49647 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 49651 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 49653 T253 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@305b8b04 name:ZooKeeperConnection Watcher:127.0.0.1:51822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 49653 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 49658 T161 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C25 name=collection1 org.apache.solr.core.SolrCore@3266f7c5 url=http://127.0.0.1:58508/collection1 node=127.0.0.1:58508_ C25_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:58508_, base_url=http://127.0.0.1:58508}
[junit4:junit4] 2> 49782 T222 C25 P58508 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:43365/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 49783 T222 C25 P58508 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:58508 START replicas=[http://127.0.0.1:43365/collection1/] nUpdates=100
[junit4:junit4] 2> 49784 T222 C25 P58508 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 49784 T222 C25 P58508 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 49784 T222 C25 P58508 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 49785 T222 C25 P58508 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 49785 T222 C25 P58508 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 49786 T222 C25 P58508 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:43365/collection1/. core=collection1
[junit4:junit4] 2> 49786 T222 C25 P58508 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C26 name=collection1 org.apache.solr.core.SolrCore@3e59c0b url=http://127.0.0.1:43365/collection1 node=127.0.0.1:43365_ C26_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:43365_, base_url=http://127.0.0.1:43365, leader=true}
[junit4:junit4] 2> 49789 T198 C26 P43365 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 49804 T197 C26 P43365 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 49806 T197 C26 P43365 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f5fb626 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7fd4880e),segFN=segments_1,generation=1}
[junit4:junit4] 2> 49807 T197 C26 P43365 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 49807 T197 C26 P43365 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 49808 T197 C26 P43365 oass.SolrIndexSearcher.<init> Opening Searcher@7b7767ee realtime
[junit4:junit4] 2> 49808 T197 C26 P43365 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 49809 T197 C26 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 15
[junit4:junit4] 2> 49832 T222 C25 P58508 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 49833 T222 C25 P58508 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4] 2> 49840 T222 C25 P58508 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 49845 T199 C26 P43365 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 49845 T199 C26 P43365 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 49846 T222 C25 P58508 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 49847 T222 C25 P58508 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 49847 T222 C25 P58508 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 49847 T222 C25 P58508 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 49849 T222 C25 P58508 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 50001 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 50002 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55874_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55874"}
[junit4:junit4] 2> 50007 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:58508_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:58508"}
[junit4:junit4] 2> 50013 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 50013 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 50013 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 50013 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 50013 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 50014 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 50514 T196 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 50515 T196 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node3&state=recovering&nodeName=127.0.0.1:55874_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4] 2> 50662 T161 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57393_
[junit4:junit4] 2> 50663 T161 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57393_
[junit4:junit4] 2> 50666 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 50667 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 50667 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 50666 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 50666 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 50667 T236 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 50668 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 50671 T183 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 50672 T190 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 50672 T219 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 50672 T203 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 50672 T253 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 50676 T255 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 50677 T255 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 50678 T255 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4] 2> 51527 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 51528 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:57393_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57393"}
[junit4:junit4] 2> 51528 T184 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 51529 T184 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 51534 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 51534 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 51534 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 51534 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 51534 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 51535 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 51679 T255 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4] 2> 51679 T255 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881/collection1
[junit4:junit4] 2> 51679 T255 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 51681 T255 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 51681 T255 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 51683 T255 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881/collection1/'
[junit4:junit4] 2> 51684 T255 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881/collection1/lib/README' to classloader
[junit4:junit4] 2> 51685 T255 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 51724 T255 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 51754 T255 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 51756 T255 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 51773 T255 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 52137 T255 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 52143 T255 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 52145 T255 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 52182 T255 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 52190 T255 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 52195 T255 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 52197 T255 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 52198 T255 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 52198 T255 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 52199 T255 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 52200 T255 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 52200 T255 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 52201 T255 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty4/
[junit4:junit4] 2> 52201 T255 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@522103ff
[junit4:junit4] 2> 52202 T255 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty4
[junit4:junit4] 2> 52202 T255 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty4/index/
[junit4:junit4] 2> 52203 T255 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 52203 T255 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty4/index
[junit4:junit4] 2> 52205 T255 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30f0e803 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7736970f),segFN=segments_1,generation=1}
[junit4:junit4] 2> 52206 T255 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 52209 T255 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 52209 T255 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 52210 T255 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 52211 T255 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 52212 T255 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 52212 T255 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 52213 T255 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 52214 T255 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 52214 T255 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 52218 T255 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 52221 T255 oass.SolrIndexSearcher.<init> Opening Searcher@e2471c7 main
[junit4:junit4] 2> 52222 T255 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 52223 T255 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 52230 T256 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e2471c7 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 52233 T255 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 52234 T255 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57393 collection:collection1 shard:shard1
[junit4:junit4] 2> 52238 T255 oasc.ZkController.register We are http://127.0.0.1:57393/collection1/ and leader is http://127.0.0.1:43365/collection1/
[junit4:junit4] 2> 52238 T255 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57393
[junit4:junit4] 2> 52238 T255 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 52239 T255 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C27 name=collection1 org.apache.solr.core.SolrCore@6c18b5da url=http://127.0.0.1:57393/collection1 node=127.0.0.1:57393_ C27_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:57393_, base_url=http://127.0.0.1:57393}
[junit4:junit4] 2> 52239 T257 C27 P57393 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 52240 T257 C27 P57393 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 52240 T255 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 52240 T257 C27 P57393 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 52240 T257 C27 P57393 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 52241 T161 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 52242 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 52242 T257 C27 P57393 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 52242 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 52246 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 52248 T199 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 52249 T161 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30 for each attempt
[junit4:junit4] 2> 52249 T161 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
[junit4:junit4] 2> 52250 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C28 name=collection1 org.apache.solr.core.SolrCore@151671f7 url=http://127.0.0.1:55874/collection1 node=127.0.0.1:55874_ C28_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:55874_, base_url=http://127.0.0.1:55874}
[junit4:junit4] 2> 52516 T239 C28 P55874 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:43365/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 52516 T239 C28 P55874 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55874 START replicas=[http://127.0.0.1:43365/collection1/] nUpdates=100
[junit4:junit4] 2> 52517 T239 C28 P55874 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 52518 T239 C28 P55874 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 52518 T239 C28 P55874 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 52519 T239 C28 P55874 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 52519 T239 C28 P55874 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 52520 T239 C28 P55874 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:43365/collection1/. core=collection1
[junit4:junit4] 2> 52520 T239 C28 P55874 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C29 name=collection1 org.apache.solr.core.SolrCore@3e59c0b url=http://127.0.0.1:43365/collection1 node=127.0.0.1:43365_ C29_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:43365_, base_url=http://127.0.0.1:43365, leader=true}
[junit4:junit4] 2> 52521 T197 C29 P43365 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 52531 T196 C29 P43365 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 52532 T196 C29 P43365 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 52533 T196 C29 P43365 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 52533 T196 C29 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 2
[junit4:junit4] 2> 52535 T239 C28 P55874 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 52535 T239 C28 P55874 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 52540 T197 C29 P43365 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 52541 T239 C28 P55874 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 52542 T239 C28 P55874 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 52542 T239 C28 P55874 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 52542 T239 C28 P55874 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 52545 T239 C28 P55874 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 53041 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 53042 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:57393_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57393"}
[junit4:junit4] 2> 53047 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55874_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55874"}
[junit4:junit4] 2> 53054 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 53054 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 53054 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 53054 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 53054 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 53055 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 53248 T199 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 53249 T199 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node4&state=recovering&nodeName=127.0.0.1:57393_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 53252 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 54255 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C27_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:57393_, base_url=http://127.0.0.1:57393}
[junit4:junit4] 2> 55250 T257 C27 P57393 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:43365/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 55250 T257 C27 P57393 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:57393 START replicas=[http://127.0.0.1:43365/collection1/] nUpdates=100
[junit4:junit4] 2> 55251 T257 C27 P57393 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 55251 T257 C27 P57393 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 55251 T257 C27 P57393 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 55251 T257 C27 P57393 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 55251 T257 C27 P57393 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 55252 T257 C27 P57393 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:43365/collection1/. core=collection1
[junit4:junit4] 2> 55252 T257 C27 P57393 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 55253 T199 C29 P43365 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 55257 T197 C29 P43365 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 55258 T197 C29 P43365 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 55258 T197 C29 P43365 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 55258 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 55258 T197 C29 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 1
[junit4:junit4] 2> 55260 T257 C27 P57393 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 55260 T257 C27 P57393 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 55262 T199 C29 P43365 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 55263 T257 C27 P57393 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 55263 T257 C27 P57393 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 55263 T257 C27 P57393 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 55264 T257 C27 P57393 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 55265 T257 C27 P57393 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 56069 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 56071 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:57393_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57393"}
[junit4:junit4] 2> 56079 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 56080 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 56080 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 56080 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 56080 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 56080 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 56261 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 56263 T161 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C30 name=collection1 org.apache.solr.core.SolrCore@8635d12 url=http://127.0.0.1:60204/collection1 node=127.0.0.1:60204_ C30_STATE=coll:control_collection core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:60204_, base_url=http://127.0.0.1:60204, leader=true}
[junit4:junit4] 2> 56273 T178 C30 P60204 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 56275 T178 C30 P60204 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5324396a lockFactory=org.apache.lucene.store.NativeFSLockFactory@59e05bf9),segFN=segments_1,generation=1}
[junit4:junit4] 2> 56276 T178 C30 P60204 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 56276 T178 C30 P60204 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 56276 T178 C30 P60204 oass.SolrIndexSearcher.<init> Opening Searcher@3eb78a85 main
[junit4:junit4] 2> 56277 T178 C30 P60204 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 56278 T187 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3eb78a85 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 56278 T178 C30 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> C27_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:57393_, base_url=http://127.0.0.1:57393}
[junit4:junit4] 2> 56298 T249 C27 P57393 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:43365/collection1/, StdNode: http://127.0.0.1:58508/collection1/, StdNode: http://127.0.0.1:55874/collection1/, StdNode: http://127.0.0.1:57393/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 56307 T246 C27 P57393 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 56307 T198 C29 P43365 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 56308 T198 C29 P43365 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 56309 T198 C29 P43365 oass.SolrIndexSearcher.<init> Opening Searcher@2ca08829 main
[junit4:junit4] 2> 56309 T246 C27 P57393 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30f0e803 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7736970f),segFN=segments_1,generation=1}
[junit4:junit4] 2> 56310 T198 C29 P43365 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 56311 T246 C27 P57393 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 56312 T205 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2ca08829 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 56312 T246 C27 P57393 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 56314 T198 C29 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> ASYNC NEW_CORE C31 name=collection1 org.apache.solr.core.SolrCore@3266f7c5 url=http://127.0.0.1:58508/collection1 node=127.0.0.1:58508_ C31_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:58508_, base_url=http://127.0.0.1:58508}
[junit4:junit4] 2> 56314 T214 C31 P58508 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 56314 T246 C27 P57393 oass.SolrIndexSearcher.<init> Opening Searcher@4a3ece73 main
[junit4:junit4] 2> C28_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:55874_, base_url=http://127.0.0.1:55874}
[junit4:junit4] 2> 56315 T232 C28 P55874 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 56316 T246 C27 P57393 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 56317 T214 C31 P58508 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1b88581a lockFactory=org.apache.lucene.store.NativeFSLockFactory@4127ea39),segFN=segments_1,generation=1}
[junit4:junit4] 2> 56318 T214 C31 P58508 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 56318 T256 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4a3ece73 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 56319 T232 C28 P55874 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4afd68fe lockFactory=org.apache.lucene.store.NativeFSLockFactory@5994519c),segFN=segments_1,generation=1}
[junit4:junit4] 2> 56320 T246 C27 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 13
[junit4:junit4] 2> 56319 T214 C31 P58508 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 56321 T232 C28 P55874 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 56322 T214 C31 P58508 oass.SolrIndexSearcher.<init> Opening Searcher@a873204 main
[junit4:junit4] 2> 56323 T232 C28 P55874 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 56323 T214 C31 P58508 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 56324 T232 C28 P55874 oass.SolrIndexSearcher.<init> Opening Searcher@5a2e8b7e main
[junit4:junit4] 2> 56325 T221 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a873204 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 56325 T232 C28 P55874 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 56326 T214 C31 P58508 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 12
[junit4:junit4] 2> 56327 T238 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5a2e8b7e main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 56328 T232 C28 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 13
[junit4:junit4] 2> 56330 T249 C27 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 33
[junit4:junit4] 2> 56332 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 56338 T197 C29 P43365 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 56343 T215 C31 P58508 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 56348 T230 C28 P55874 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 56353 T248 C27 P57393 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 58360 T179 C30 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1439074324554186752)} 0 3
[junit4:junit4] 2> 58371 T231 C28 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1439074324562575360&update.from=http://127.0.0.1:43365/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439074324562575360)} 0 2
[junit4:junit4] 2> 58371 T212 C31 P58508 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1439074324562575360&update.from=http://127.0.0.1:43365/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439074324562575360)} 0 1
[junit4:junit4] 2> 58371 T248 C27 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1439074324562575360&update.from=http://127.0.0.1:43365/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439074324562575360)} 0 1
[junit4:junit4] 2> 58372 T197 C29 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1439074324562575360)} 0 7
[junit4:junit4] 2> 58373 T229 C28 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 11
[junit4:junit4] 2> 58378 T178 C30 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1439074324573061120)]} 0 3
[junit4:junit4] 2> 58395 T249 C27 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439074324584595456)]} 0 3
[junit4:junit4] 2> 58395 T232 C28 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439074324584595456)]} 0 3
[junit4:junit4] 2> 58396 T213 C31 P58508 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439074324584595456)]} 0 4
[junit4:junit4] 2> 58397 T196 C29 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:57393/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1439074324584595456)]} 0 11
[junit4:junit4] 2> 58398 T246 C27 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0]} 0 17
[junit4:junit4] 2> 58403 T177 C30 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1439074324599275520)]} 0 3
[junit4:junit4] 2> 58416 T230 C28 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439074324603469824)]} 0 2
[junit4:junit4] 2> 58418 T214 C31 P58508 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439074324603469824)]} 0 4
[junit4:junit4] 2> 58417 T247 C27 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439074324603469824)]} 0 3
[junit4:junit4] 2> 58419 T199 C29 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1439074324603469824)]} 0 14
[junit4:junit4] 2> ASYNC NEW_CORE C32 name=collection1 org.apache.solr.core.SolrCore@8635d12 url=http://127.0.0.1:60204/collection1 node=127.0.0.1:60204_ C32_STATE=coll:control_collection core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:60204_, base_url=http://127.0.0.1:60204, leader=true}
[junit4:junit4] 2> 58433 T176 C32 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1439074324631781376)]} 0 2
[junit4:junit4] 2> ASYNC NEW_CORE C33 name=collection1 org.apache.solr.core.SolrCore@3e59c0b url=http://127.0.0.1:43365/collection1 node=127.0.0.1:43365_ C33_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:43365_, base_url=http://127.0.0.1:43365, leader=true}
[junit4:junit4] 2> 58440 T198 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:58508/collection1/]
[junit4:junit4] 2> 58440 T198 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:58508/collection1/ against:[http://127.0.0.1:58508/collection1/] result:true
[junit4:junit4] 2> 58440 T198 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:55874/collection1/ against:[http://127.0.0.1:58508/collection1/] result:false
[junit4:junit4] 2> 58441 T198 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:57393/collection1/ against:[http://127.0.0.1:58508/collection1/] result:false
[junit4:junit4] 2> ASYNC NEW_CORE C34 name=collection1 org.apache.solr.core.SolrCore@151671f7 url=http://127.0.0.1:55874/collection1 node=127.0.0.1:55874_ C34_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:55874_, base_url=http://127.0.0.1:55874}
[junit4:junit4] 2> 58447 T231 C34 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[2 (1439074324641218560)]} 0 2
[junit4:junit4] 2> ASYNC NEW_CORE C35 name=collection1 org.apache.solr.core.SolrCore@6c18b5da url=http://127.0.0.1:57393/collection1 node=127.0.0.1:57393_ C35_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:57393_, base_url=http://127.0.0.1:57393}
[junit4:junit4] 2> 58447 T249 C35 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[2 (1439074324641218560)]} 0 2
[junit4:junit4] 2> 58448 T198 C33 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:57393/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[2 (1439074324641218560)]} 0 9
[junit4:junit4] 2> 58449 T248 C35 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[2]} 0 13
[junit4:junit4] 2> 58455 T179 C32 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1439074324652752896)]} 0 3
[junit4:junit4] 2> 58458 T197 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:58508/collection1/, http://127.0.0.1:55874/collection1/]
[junit4:junit4] 2> 58459 T197 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:58508/collection1/ against:[http://127.0.0.1:58508/collection1/, http://127.0.0.1:55874/collection1/] result:true
[junit4:junit4] 2> 58459 T197 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:55874/collection1/ against:[http://127.0.0.1:58508/collection1/, http://127.0.0.1:55874/collection1/] result:true
[junit4:junit4] 2> 58459 T197 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:57393/collection1/ against:[http://127.0.0.1:58508/collection1/, http://127.0.0.1:55874/collection1/] result:false
[junit4:junit4] 2> 58466 T246 C35 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[3 (1439074324661141504)]} 0 2
[junit4:junit4] 2> 58467 T197 C33 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&test.distrib.skip.servers=http://127.0.0.1:55874/collection1/&wt=javabin&version=2} {add=[3 (1439074324661141504)]} 0 9
[junit4:junit4] 2> 58470 T178 C32 P60204 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 58479 T178 C32 P60204 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5324396a lockFactory=org.apache.lucene.store.NativeFSLockFactory@59e05bf9),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5324396a lockFactory=org.apache.lucene.store.NativeFSLockFactory@59e05bf9),segFN=segments_2,generation=2}
[junit4:junit4] 2> 58480 T178 C32 P60204 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 58487 T178 C32 P60204 oass.SolrIndexSearcher.<init> Opening Searcher@3043b1cc main
[junit4:junit4] 2> 58488 T178 C32 P60204 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 58489 T187 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3043b1cc main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 58490 T178 C32 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 21
[junit4:junit4] 2> 58493 T229 C34 P55874 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:43365/collection1/, StdNode: http://127.0.0.1:58508/collection1/, StdNode: http://127.0.0.1:55874/collection1/, StdNode: http://127.0.0.1:57393/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> ASYNC NEW_CORE C36 name=collection1 org.apache.solr.core.SolrCore@3266f7c5 url=http://127.0.0.1:58508/collection1 node=127.0.0.1:58508_ C36_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:58508_, base_url=http://127.0.0.1:58508}
[junit4:junit4] 2> 58496 T215 C36 P58508 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 58497 T247 C35 P57393 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 58497 T232 C34 P55874 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 58497 T196 C33 P43365 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 58505 T215 C36 P58508 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1b88581a lockFactory=org.apache.lucene.store.NativeFSLockFactory@4127ea39),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1b88581a lockFactory=org.apache.lucene.store.NativeFSLockFactory@4127ea39),segFN=segments_2,generation=2}
[junit4:junit4] 2> 58506 T215 C36 P58508 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 58507 T232 C34 P55874 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4afd68fe lockFactory=org.apache.lucene.store.NativeFSLockFactory@5994519c),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4afd68fe lockFactory=org.apache.lucene.store.NativeFSLockFactory@5994519c),segFN=segments_2,generation=2}
[junit4:junit4] 2> 58508 T232 C34 P55874 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 58507 T247 C35 P57393 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30f0e803 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7736970f),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30f0e803 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7736970f),segFN=segments_2,generation=2}
[junit4:junit4] 2> 58509 T247 C35 P57393 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 58510 T215 C36 P58508 oass.SolrIndexSearcher.<init> Opening Searcher@42ccfd11 main
[junit4:junit4] 2> 58511 T215 C36 P58508 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 58511 T221 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@42ccfd11 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c2)}
[junit4:junit4] 2> 58512 T247 C35 P57393 oass.SolrIndexSearcher.<init> Opening Searcher@4e383c54 main
[junit4:junit4] 2> 58512 T215 C36 P58508 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 16
[junit4:junit4] 2> 58512 T247 C35 P57393 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 58513 T256 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4e383c54 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 58514 T232 C34 P55874 oass.SolrIndexSearcher.<init> Opening Searcher@7ca99a28 main
[junit4:junit4] 2> 58514 T247 C35 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 18
[junit4:junit4] 2> 58514 T232 C34 P55874 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 58515 T238 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7ca99a28 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c3)}
[junit4:junit4] 2> 58516 T232 C34 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 20
[junit4:junit4] 2> 58519 T196 C33 P43365 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f5fb626 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7fd4880e),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f5fb626 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7fd4880e),segFN=segments_2,generation=2}
[junit4:junit4] 2> 58519 T196 C33 P43365 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 58523 T196 C33 P43365 oass.SolrIndexSearcher.<init> Opening Searcher@7c7686f8 main
[junit4:junit4] 2> 58523 T196 C33 P43365 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 58524 T205 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7c7686f8 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 58525 T196 C33 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 29
[junit4:junit4] 2> 58526 T229 C34 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 33
[junit4:junit4] 2> 58526 T161 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):330
[junit4:junit4] 2> 58527 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 58528 T161 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 58531 T199 C33 P43365 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 58534 T212 C36 P58508 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0
[junit4:junit4] 2> 58538 T230 C34 P55874 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=3 status=0 QTime=1
[junit4:junit4] 2> 58541 T249 C35 P57393 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 58543 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 58549 T231 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4] 2> 58550 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 58555 T198 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4] 2> 58556 T198 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43365/collection1/
[junit4:junit4] 2> 58556 T198 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:43365 START replicas=[http://127.0.0.1:58508/collection1/, http://127.0.0.1:55874/collection1/, http://127.0.0.1:57393/collection1/] nUpdates=100
[junit4:junit4] 2> 58560 T229 C34 P55874 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 58560 T246 C35 P57393 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 58560 T213 C36 P58508 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 58560 T198 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:43365 Received 4 versions from 127.0.0.1:55874/collection1/
[junit4:junit4] 2> 58561 T198 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:43365 Our versions are newer. ourLowThreshold=1439074324562575360 otherHigh=1439074324641218560
[junit4:junit4] 2> 58561 T198 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:43365 Received 5 versions from 127.0.0.1:57393/collection1/
[junit4:junit4] 2> 58562 T198 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:43365 Our versions are newer. ourLowThreshold=1439074324562575360 otherHigh=1439074324641218560
[junit4:junit4] 2> 58562 T198 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:43365 Received 3 versions from 127.0.0.1:58508/collection1/
[junit4:junit4] 2> 58562 T198 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:43365 Our versions are newer. ourLowThreshold=1439074324562575360 otherHigh=1439074324603469824
[junit4:junit4] 2> 58563 T198 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:43365 DONE. sync succeeded
[junit4:junit4] 2> 58563 T198 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 58563 T198 oasc.SyncStrategy.syncToMe http://127.0.0.1:43365/collection1/: try and ask http://127.0.0.1:58508/collection1/ to sync
[junit4:junit4] 2> 58564 T198 oasc.SyncStrategy.syncToMe http://127.0.0.1:43365/collection1/: try and ask http://127.0.0.1:55874/collection1/ to sync
[junit4:junit4] 2> 58564 T198 oasc.SyncStrategy.syncToMe http://127.0.0.1:43365/collection1/: try and ask http://127.0.0.1:57393/collection1/ to sync
[junit4:junit4] 2> 58566 T214 C36 P58508 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:58508 START replicas=[http://127.0.0.1:43365/collection1/] nUpdates=100
[junit4:junit4] 2> 58567 T249 C35 P57393 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:57393 START replicas=[http://127.0.0.1:43365/collection1/] nUpdates=100
[junit4:junit4] 2> 58568 T232 C34 P55874 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55874 START replicas=[http://127.0.0.1:43365/collection1/] nUpdates=100
[junit4:junit4] 2> 58570 T199 C33 P43365 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 58571 T196 C33 P43365 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 58571 T214 C36 P58508 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:58508 Received 5 versions from 127.0.0.1:43365/collection1/
[junit4:junit4] 2> 58572 T249 C35 P57393 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:57393 Received 5 versions from 127.0.0.1:43365/collection1/
[junit4:junit4] 2> 58571 T197 C33 P43365 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 58572 T249 C35 P57393 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:57393 Our versions are newer. ourLowThreshold=1439074324562575360 otherHigh=1439074324641218560
[junit4:junit4] 2> 58572 T214 C36 P58508 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:58508 Requesting updates from 127.0.0.1:43365/collection1/n=2 versions=[1439074324661141504, 1439074324641218560]
[junit4:junit4] 2> 58573 T232 C34 P55874 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:55874 Received 5 versions from 127.0.0.1:43365/collection1/
[junit4:junit4] 2> 58573 T249 C35 P57393 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:57393 DONE. sync succeeded
[junit4:junit4] 2> 58574 T232 C34 P55874 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:55874 Requesting updates from 127.0.0.1:43365/collection1/n=1 versions=[1439074324661141504]
[junit4:junit4] 2> 58575 T249 C35 P57393 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:43365/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=8
[junit4:junit4] 2> 58576 T198 oasc.SyncStrategy.syncToMe http://127.0.0.1:43365/collection1/: sync completed with http://127.0.0.1:57393/collection1/
[junit4:junit4] 2> 58576 T199 C33 P43365 oasc.SolrCore.execute [collection1] webapp= path=/get params={distrib=false&getUpdates=1439074324661141504,1439074324641218560&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 58578 T196 C33 P43365 oasc.SolrCore.execute [collection1] webapp= path=/get params={distrib=false&getUpdates=1439074324661141504&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 58583 T232 C34 P55874 oasup.LogUpdateProcessor.finish [collection1] {add=[3 (1439074324661141504)]} 0 4
[junit4:junit4] 2> 58583 T232 C34 P55874 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55874 DONE. sync succeeded
[junit4:junit4] 2> 58584 T232 C34 P55874 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:43365/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=16
[junit4:junit4] 2> 58584 T198 oasc.SyncStrategy.syncToMe http://127.0.0.1:43365/collection1/: sync completed with http://127.0.0.1:55874/collection1/
[junit4:junit4] 2> 58586 T214 C36 P58508 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1439074324641218560), 3 (1439074324661141504)]} 0 9
[junit4:junit4] 2> 58586 T214 C36 P58508 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:58508 DONE. sync succeeded
[junit4:junit4] 2> 58587 T214 C36 P58508 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:43365/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=20
[junit4:junit4] 2> 58587 T198 oasc.SyncStrategy.syncToMe http://127.0.0.1:43365/collection1/: sync completed with http://127.0.0.1:58508/collection1/
[junit4:junit4] 2> 58588 T198 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=33
[junit4:junit4] 2> 58588 T231 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=40
[junit4:junit4] 2> 58589 T161 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 58589 T161 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 58590 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 58591 T161 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 58594 T177 C32 P60204 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 58595 T177 C32 P60204 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 58595 T177 C32 P60204 oass.SolrIndexSearcher.<init> Opening Searcher@6122ccff main
[junit4:junit4] 2> 58596 T177 C32 P60204 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 58597 T187 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6122ccff main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 58597 T177 C32 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4] 2> 58600 T215 C36 P58508 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:43365/collection1/, StdNode: http://127.0.0.1:58508/collection1/, StdNode: http://127.0.0.1:55874/collection1/, StdNode: http://127.0.0.1:57393/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 58604 T230 C34 P55874 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 58604 T197 C33 P43365 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 58606 T197 C33 P43365 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 58606 T246 C35 P57393 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 58607 T197 C33 P43365 oass.SolrIndexSearcher.<init> Opening Searcher@5923ec42 main
[junit4:junit4] 2> 58606 T213 C36 P58508 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 58608 T197 C33 P43365 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 58607 T246 C35 P57393 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 58610 T205 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5923ec42 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 58611 T197 C33 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 2> 58611 T246 C35 P57393 oass.SolrIndexSearcher.<init> Opening Searcher@34499143 main
[junit4:junit4] 2> 58612 T246 C35 P57393 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 58616 T256 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@34499143 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4] 2> 58617 T246 C35 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 12
[junit4:junit4] 2> 58623 T230 C34 P55874 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4afd68fe lockFactory=org.apache.lucene.store.NativeFSLockFactory@5994519c),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4afd68fe lockFactory=org.apache.lucene.store.NativeFSLockFactory@5994519c),segFN=segments_3,generation=3}
[junit4:junit4] 2> 58623 T213 C36 P58508 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1b88581a lockFactory=org.apache.lucene.store.NativeFSLockFactory@4127ea39),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1b88581a lockFactory=org.apache.lucene.store.NativeFSLockFactory@4127ea39),segFN=segments_3,generation=3}
[junit4:junit4] 2> 58624 T213 C36 P58508 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 58623 T230 C34 P55874 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 58628 T213 C36 P58508 oass.SolrIndexSearcher.<init> Opening Searcher@7754fdc4 main
[junit4:junit4] 2> 58628 T230 C34 P55874 oass.SolrIndexSearcher.<init> Opening Searcher@6ce08c10 main
[junit4:junit4] 2> 58630 T213 C36 P58508 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 58631 T221 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7754fdc4 main{StandardDirectoryReader(segments_3:5:nrt _0(5.0):c2 _1(5.0):c2)}
[junit4:junit4] 2> 58631 T213 C36 P58508 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 25
[junit4:junit4] 2> 58632 T238 oasc.JmxMonitoredMap.put WARN Failed to register info bean: searcher javax.management.InstanceAlreadyExistsException: solr/collection1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4] 2> at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:305)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1844)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1721)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2>
[junit4:junit4] 2> 58632 T230 C34 P55874 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 58633 T238 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6ce08c10 main{StandardDirectoryReader(segments_3:5:nrt _0(5.0):c3 _1(5.0):c1)}
[junit4:junit4] 2> 58634 T230 C34 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 30
[junit4:junit4] 2> 58635 T215 C36 P58508 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 35
[junit4:junit4] 2> 58637 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 58640 T196 C33 P43365 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 58644 T214 C36 P58508 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 58647 T232 C34 P55874 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 58650 T247 C35 P57393 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 60652 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> check const of shard1
[junit4:junit4] 2> client0
[junit4:junit4] 2> PROPS:core_node1:{"state":"active","core":"collection1","node_name":"127.0.0.1:43365_","base_url":"http://127.0.0.1:43365","leader":"true"}
[junit4:junit4] 2> 60657 T197 C33 P43365 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client1
[junit4:junit4] 2> PROPS:core_node2:{"state":"active","core":"collection1","node_name":"127.0.0.1:58508_","base_url":"http://127.0.0.1:58508"}
[junit4:junit4] 2> 60662 T215 C36 P58508 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client2
[junit4:junit4] 2> PROPS:core_node3:{"state":"active","core":"collection1","node_name":"127.0.0.1:55874_","base_url":"http://127.0.0.1:55874"}
[junit4:junit4] 2> 60666 T230 C34 P55874 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client3
[junit4:junit4] 2> PROPS:core_node4:{"state":"active","core":"collection1","node_name":"127.0.0.1:57393_","base_url":"http://127.0.0.1:57393"}
[junit4:junit4] 2> 60670 T247 C35 P57393 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> 60675 T231 C34 P55874 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&wt=javabin&version=2} hits=4 status=0 QTime=1
[junit4:junit4] 2> 60686 T178 C32 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1439074326991077376)]} 0 5
[junit4:junit4] 2> 60693 T196 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:58508/collection1/]
[junit4:junit4] 2> 60694 T196 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:58508/collection1/ against:[http://127.0.0.1:58508/collection1/] result:true
[junit4:junit4] 2> 60694 T196 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:55874/collection1/ against:[http://127.0.0.1:58508/collection1/] result:false
[junit4:junit4] 2> 60694 T196 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:57393/collection1/ against:[http://127.0.0.1:58508/collection1/] result:false
[junit4:junit4] 2> 60706 T232 C34 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[4 (1439074327004708864)]} 0 7
[junit4:junit4] 2> 60706 T248 C35 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[4 (1439074327004708864)]} 0 7
[junit4:junit4] 2> 60708 T196 C33 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:57393/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[4 (1439074327004708864)]} 0 15
[junit4:junit4] 2> 60709 T249 C35 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[4]} 0 19
[junit4:junit4] 2> 60710 T161 oasc.ChaosMonkey.monkeyLog monkey: kill shard! 43365
[junit4:junit4] 2> 60734 T161 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1430281418
[junit4:junit4] 2> 62101 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 62102 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:43365_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43365"}
[junit4:junit4] 2> 62102 T184 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4:junit4] 2> 62106 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 62106 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 62106 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 62106 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 62106 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 62106 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 63740 T161 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 63741 T161 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 63745 T161 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@3e59c0b
[junit4:junit4] 2> 63756 T161 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=5,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 63757 T161 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 63758 T161 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 63758 T161 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> C33_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:43365_, base_url=http://127.0.0.1:43365, leader=true}
[junit4:junit4] 2> 63769 T161 C33 P43365 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f5fb626 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7fd4880e),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f5fb626 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7fd4880e),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f5fb626 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7fd4880e),segFN=segments_3,generation=3}
[junit4:junit4] 2> 63770 T161 C33 P43365 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 63773 T161 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 63774 T161 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 63774 T161 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1/index;done=false>>]
[junit4:junit4] 2> 63775 T161 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1/index
[junit4:junit4] 2> 63776 T161 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1;done=false>>]
[junit4:junit4] 2> 63776 T161 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1
[junit4:junit4] 2> 63780 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 63781 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 63781 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 63781 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 63785 T161 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 63785 T203 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 63786 T203 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 63785 T236 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 63787 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 63790 T183 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 63790 T219 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4] 2> 63791 T190 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 63791 T253 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 63795 T219 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 63796 T219 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 63796 T219 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 63796 T219 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:58508/collection1/
[junit4:junit4] 2> 63797 T219 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:58508 START replicas=[http://127.0.0.1:43365/collection1/, http://127.0.0.1:55874/collection1/, http://127.0.0.1:57393/collection1/] nUpdates=100
[junit4:junit4] 2> 63801 T219 oasu.PeerSync.handleResponse WARN PeerSync: core=collection1 url=http://127.0.0.1:58508 couldn't connect to http://127.0.0.1:43365/collection1/, counting as success
[junit4:junit4] 2> 63802 T232 C34 P55874 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 63802 T246 C35 P57393 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 63804 T219 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:58508 Received 6 versions from 127.0.0.1:55874/collection1/
[junit4:junit4] 2> 63804 T219 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:58508 Requesting updates from 127.0.0.1:55874/collection1/n=1 versions=[1439074327004708864]
[junit4:junit4] 2> 63805 T219 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:58508 Received 6 versions from 127.0.0.1:57393/collection1/
[junit4:junit4] 2> 63805 T219 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:58508 Our versions are newer. ourLowThreshold=1439074324562575360 otherHigh=1439074324661141504
[junit4:junit4] 2> 63808 T230 C34 P55874 oasc.SolrCore.execute [collection1] webapp= path=/get params={distrib=false&getUpdates=1439074327004708864&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 63815 T219 oasup.LogUpdateProcessor.finish [collection1] {add=[4 (1439074327004708864)]} 0 6
[junit4:junit4] 2> 63815 T219 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:58508 DONE. sync succeeded
[junit4:junit4] 2> 63815 T219 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 63816 T219 oasc.SyncStrategy.syncToMe http://127.0.0.1:58508/collection1/: try and ask http://127.0.0.1:43365/collection1/ to sync
[junit4:junit4] 2> 63816 T219 oasc.SyncStrategy.syncToMe http://127.0.0.1:58508/collection1/: try and ask http://127.0.0.1:55874/collection1/ to sync
[junit4:junit4] 2> 63817 T219 oasc.SyncStrategy.syncToMe http://127.0.0.1:58508/collection1/: try and ask http://127.0.0.1:57393/collection1/ to sync
[junit4:junit4] 2> 63818 T219 oasc.SolrException.log ERROR Sync request error: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:43365/collection1
[junit4:junit4] 2> 63819 T219 oasc.SyncStrategy.syncToMe http://127.0.0.1:58508/collection1/: Sync failed - asking replica (http://127.0.0.1:43365/collection1/) to recover.
[junit4:junit4] 2> 63821 T231 C34 P55874 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55874 START replicas=[http://127.0.0.1:58508/collection1/] nUpdates=100
[junit4:junit4] 2> 63821 T248 C35 P57393 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:57393 START replicas=[http://127.0.0.1:58508/collection1/] nUpdates=100
[junit4:junit4] 2> 63845 T293 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 63846 T161 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 43365
[junit4:junit4] 2> ASYNC NEW_CORE C37 name=collection1 org.apache.solr.core.SolrCore@3266f7c5 url=http://127.0.0.1:58508/collection1 node=127.0.0.1:58508_ C37_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:58508_, base_url=http://127.0.0.1:58508}
[junit4:junit4] 2> 63848 T213 C37 P58508 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 63848 T215 C37 P58508 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> ASYNC NEW_CORE C38 name=collection1 org.apache.solr.core.SolrCore@151671f7 url=http://127.0.0.1:55874/collection1 node=127.0.0.1:55874_ C38_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:55874_, base_url=http://127.0.0.1:55874}
[junit4:junit4] 2> 63849 T231 C38 P55874 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:55874 Received 6 versions from 127.0.0.1:58508/collection1/
[junit4:junit4] 2> ASYNC NEW_CORE C39 name=collection1 org.apache.solr.core.SolrCore@6c18b5da url=http://127.0.0.1:57393/collection1 node=127.0.0.1:57393_ C39_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:57393_, base_url=http://127.0.0.1:57393}
[junit4:junit4] 2> 63850 T248 C39 P57393 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:57393 Received 6 versions from 127.0.0.1:58508/collection1/
[junit4:junit4] 2> 63850 T231 C38 P55874 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:55874 Our versions are newer. ourLowThreshold=1439074324584595456 otherHigh=1439074324661141504
[junit4:junit4] 2> 63851 T248 C39 P57393 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:57393 Our versions are newer. ourLowThreshold=1439074324584595456 otherHigh=1439074324661141504
[junit4:junit4] 2> 63852 T248 C39 P57393 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:57393 DONE. sync succeeded
[junit4:junit4] 2> 63851 T231 C38 P55874 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55874 DONE. sync succeeded
[junit4:junit4] 2> 63853 T248 C39 P57393 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:58508/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=32
[junit4:junit4] 2> 63853 T231 C38 P55874 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:58508/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=33
[junit4:junit4] 2> 63854 T219 oasc.SyncStrategy.syncToMe http://127.0.0.1:58508/collection1/: sync completed with http://127.0.0.1:57393/collection1/
[junit4:junit4] 2> 63855 T219 oasc.SyncStrategy.syncToMe http://127.0.0.1:58508/collection1/: sync completed with http://127.0.0.1:55874/collection1/
[junit4:junit4] 2> 63856 T219 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:58508/collection1/ shard1
[junit4:junit4] 2> 63856 T219 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 63857 T293 oasc.SolrException.log ERROR http://127.0.0.1:58508/collection1/: Could not tell a replica to recover:org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:43365
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4] 2> at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:297)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:43365 refused
[junit4:junit4] 2> at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
[junit4:junit4] 2> at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4:junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4:junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4:junit4] 2> ... 5 more
[junit4:junit4] 2> Caused by: java.net.ConnectException: Connection refused
[junit4:junit4] 2> at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4:junit4] 2> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4:junit4] 2> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4:junit4] 2> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4:junit4] 2> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[junit4:junit4] 2> at java.net.Socket.connect(Socket.java:579)
[junit4:junit4] 2> at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4:junit4] 2> at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4:junit4] 2> ... 12 more
[junit4:junit4] 2>
[junit4:junit4] 2> 63867 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 63871 T219 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 65116 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 65117 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"core_node1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:43365_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43365"}
[junit4:junit4] 2> 65130 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 65130 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 65130 T253 oascc.ZkStateReader$2.process A cluster
[...truncated too long message...]
:"127.0.0.1:43365_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:43365"},
[junit4:junit4] 1> "core_node2":{
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:58508_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:58508"},
[junit4:junit4] 1> "core_node3":{
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:55874_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:55874"},
[junit4:junit4] 1> "core_node4":{
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:57393_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:57393",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"core_node1":{
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:60204_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:60204",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:55874_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:57393_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:43365_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:60204_ (0)
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/collections (2)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89942143517917196-core_node4-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89942143517917198-core_node1-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89942143517917194-core_node3-n_0000000005 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (1)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:57393_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:57393"}
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89942143517917187-core_node1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:60204_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:60204"}
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/election (4)
[junit4:junit4] 1> /solr/overseer_elect/election/89942143517917187-127.0.0.1:60204_-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89942143517917198-127.0.0.1:43365_-n_0000000005 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89942143517917196-127.0.0.1:57393_-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89942143517917194-127.0.0.1:55874_-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89942143517917187-127.0.0.1:60204_-n_0000000000"}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=EE579D43B894FBE1 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=iw -Dtests.timezone=Asia/Pontianak -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE 106s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:43365/collection1lastClient and got 256 from http://127.0.0.1:55874/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([EE579D43B894FBE1:6FB1135BCFCB9BDD]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1045)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:237)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 137676 T161 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 106362 T160 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
[junit4:junit4] 2> 137799 T236 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 137800 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 137800 T236 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=PostingsFormat(name=Direct), text=PostingsFormat(name=Memory doPackFST= true), _version_=PostingsFormat(name=Direct), rnd_b=PostingsFormat(name=Memory doPackFST= false), intDefault=PostingsFormat(name=Direct), timestamp=PostingsFormat(name=Direct), id=PostingsFormat(name=Memory doPackFST= false), a_t=PostingsFormat(name=Direct), range_facet_sl=PostingsFormat(name=Memory doPackFST= true), range_facet_si=PostingsFormat(name=SimpleText), other_tl1=PostingsFormat(name=Direct), multiDefault=PostingsFormat(name=SimpleText), a_si=PostingsFormat(name=SimpleText)}, docValues:{timestamp=DocValuesFormat(name=SimpleText)}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=iw, timezone=Asia/Pontianak
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=8,threads=1,free=53343576,total=126681088
[junit4:junit4] 2> NOTE: All tests run in this JVM: [ScriptEngineTest, DirectUpdateHandlerOptimizeTest, ParsingFieldUpdateProcessorsTest, TestCharFilters, TestSchemaVersionResource, RAMDirectoryFactoryTest, TestLRUCache, TestAddFieldRealTimeGet, TestDynamicFieldCollectionResource, TestSolrQueryParser, PeerSyncTest, CacheHeaderTest, UpdateRequestProcessorFactoryTest, SoftAutoCommitTest, PrimitiveFieldTypeTest, TestFastLRUCache, AssignTest, SyncSliceTest]
[junit4:junit4] Completed on J0 in 107.40s, 1 test, 1 failure <<< FAILURES!
[...truncated 874 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:190: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:443: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1245: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:889: There were test failures: 312 suites, 1316 tests, 1 failure, 20 ignored (13 assumptions)
Total time: 37 minutes 56 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_25 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure