You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/27 04:51:33 UTC
[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0-ea-b94) - Build #
6256 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6256/
Java: 64bit/jdk1.8.0-ea-b94 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:36509/qs/collection1lastClient and got 246 from http://127.0.0.1:51661/qs/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:36509/qs/collection1lastClient and got 246 from http://127.0.0.1:51661/qs/collection1
at __randomizedtesting.SeedInfo.seed([9671151141C19F1F:17979B09369EFF23]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:491)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9513 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4] 2> 96251 T499 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /qs/
[junit4:junit4] 2> 96256 T499 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1372300441530
[junit4:junit4] 2> 96257 T499 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 96258 T500 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 96358 T499 oasc.ZkTestServer.run start zk server on port:50348
[junit4:junit4] 2> 96359 T499 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 96362 T506 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10ee50b7 name:ZooKeeperConnection Watcher:127.0.0.1:50348 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 96362 T499 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 96363 T499 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 96366 T499 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 96375 T508 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e29bcda name:ZooKeeperConnection Watcher:127.0.0.1:50348/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 96375 T499 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 96375 T499 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 96378 T499 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 96380 T499 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 96382 T499 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 96384 T499 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 96391 T499 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 96395 T499 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 96396 T499 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 96399 T499 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 96400 T499 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 96402 T499 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 96402 T499 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 96404 T499 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 96405 T499 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 96407 T499 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 96408 T499 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 96410 T499 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 96410 T499 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 96412 T499 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 96413 T499 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 96415 T499 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 96416 T499 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 96502 T499 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 96505 T499 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49822
[junit4:junit4] 2> 96506 T499 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 96506 T499 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 96506 T499 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372300441692
[junit4:junit4] 2> 96507 T499 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372300441692/solr.xml
[junit4:junit4] 2> 96507 T499 oasc.CoreContainer.<init> New CoreContainer 2119178982
[junit4:junit4] 2> 96507 T499 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372300441692/'
[junit4:junit4] 2> 96508 T499 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372300441692/'
[junit4:junit4] 2> 96580 T499 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 96581 T499 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 96581 T499 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 96582 T499 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 96582 T499 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 96583 T499 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 96583 T499 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 96583 T499 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 96584 T499 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 96584 T499 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 96591 T499 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 96592 T499 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50348/solr
[junit4:junit4] 2> 96601 T499 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 96603 T499 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 96605 T519 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28b61d7f name:ZooKeeperConnection Watcher:127.0.0.1:50348 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 96605 T499 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 96609 T499 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 96616 T499 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 96619 T521 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6effca54 name:ZooKeeperConnection Watcher:127.0.0.1:50348/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 96619 T499 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 96625 T499 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 96628 T499 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 96633 T499 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 96636 T499 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49822_qs
[junit4:junit4] 2> 96637 T499 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49822_qs
[junit4:junit4] 2> 96650 T499 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 96664 T499 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 96669 T499 oasc.Overseer.start Overseer (id=89935081736306691-127.0.0.1:49822_qs-n_0000000000) starting
[junit4:junit4] 2> 96672 T499 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 96683 T523 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 96683 T499 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 96690 T499 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 96692 T499 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 96694 T522 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 96698 T524 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 96698 T524 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 98198 T522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 98198 T522 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49822_qs",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49822/qs"}
[junit4:junit4] 2> 98199 T522 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 98199 T522 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 98203 T521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 98700 T524 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372300441692/collection1
[junit4:junit4] 2> 98700 T524 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 98701 T524 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 98702 T524 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 98703 T524 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372300441692/collection1/'
[junit4:junit4] 2> 98704 T524 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1372300441692/collection1/lib/README' to classloader
[junit4:junit4] 2> 98705 T524 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1372300441692/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 98737 T524 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 98773 T524 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 98779 T524 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 98788 T524 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 99197 T524 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 99198 T524 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 99198 T524 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 99204 T524 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 99207 T524 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 99225 T524 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 99229 T524 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 99232 T524 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 99233 T524 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 99233 T524 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 99234 T524 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 99235 T524 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 99235 T524 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 99235 T524 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 99242 T524 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372300441692/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372300441530/control/data/
[junit4:junit4] 2> 99243 T524 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@135776f4
[junit4:junit4] 2> 99243 T524 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 99244 T524 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/control/data
[junit4:junit4] 2> 99244 T524 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372300441530/control/data/index/
[junit4:junit4] 2> 99245 T524 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372300441530/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 99245 T524 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/control/data/index
[junit4:junit4] 2> 99260 T524 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 99261 T524 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 99283 T524 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 99283 T524 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 99284 T524 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 99285 T524 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 99285 T524 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 99286 T524 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 99286 T524 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 99287 T524 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 99287 T524 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 99289 T524 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 99292 T524 oass.SolrIndexSearcher.<init> Opening Searcher@7e58f793 main
[junit4:junit4] 2> 99292 T524 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 99292 T524 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 99296 T525 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7e58f793 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 99302 T524 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 99302 T524 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49822/qs collection:control_collection shard:shard1
[junit4:junit4] 2> 99306 T524 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 99320 T524 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 99325 T524 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 99327 T524 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 99328 T524 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 99328 T524 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49822/qs/collection1/
[junit4:junit4] 2> 99328 T524 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 99328 T524 oasc.SyncStrategy.syncToMe http://127.0.0.1:49822/qs/collection1/ has no replicas
[junit4:junit4] 2> 99329 T524 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49822/qs/collection1/
[junit4:junit4] 2> 99329 T524 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 99706 T522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 99713 T521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 99734 T524 oasc.ZkController.register We are http://127.0.0.1:49822/qs/collection1/ and leader is http://127.0.0.1:49822/qs/collection1/
[junit4:junit4] 2> 99734 T524 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49822/qs
[junit4:junit4] 2> 99735 T524 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 99735 T524 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 99735 T524 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 99737 T524 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 99739 T499 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 99739 T499 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 99740 T499 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 99746 T499 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 99749 T499 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 99751 T528 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@723c5a3b name:ZooKeeperConnection Watcher:127.0.0.1:50348/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 99751 T499 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 99753 T499 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 99756 T499 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 99858 T499 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 99862 T499 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36509
[junit4:junit4] 2> 99862 T499 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 99863 T499 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 99863 T499 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372300445030
[junit4:junit4] 2> 99864 T499 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372300445030/solr.xml
[junit4:junit4] 2> 99864 T499 oasc.CoreContainer.<init> New CoreContainer 1045900807
[junit4:junit4] 2> 99865 T499 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372300445030/'
[junit4:junit4] 2> 99865 T499 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372300445030/'
[junit4:junit4] 2> 99940 T499 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 99940 T499 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 99941 T499 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 99941 T499 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 99942 T499 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 99942 T499 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 99942 T499 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 99943 T499 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 99943 T499 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 99944 T499 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 99950 T499 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 99951 T499 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50348/solr
[junit4:junit4] 2> 99952 T499 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 99953 T499 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 99955 T539 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10b242af name:ZooKeeperConnection Watcher:127.0.0.1:50348 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 99955 T499 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 99957 T499 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 99964 T499 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 99966 T541 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7dbe17a4 name:ZooKeeperConnection Watcher:127.0.0.1:50348/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 99966 T499 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 99969 T499 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 100972 T499 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36509_qs
[junit4:junit4] 2> 100973 T499 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36509_qs
[junit4:junit4] 2> 100976 T528 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 100976 T521 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 100976 T541 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 100976 T521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 100981 T542 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 100982 T542 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 101221 T522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 101222 T522 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:49822_qs_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49822_qs",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49822/qs"}
[junit4:junit4] 2> 101225 T522 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:36509_qs",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:36509/qs"}
[junit4:junit4] 2> 101226 T522 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 101226 T522 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 101230 T541 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 101230 T528 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 101231 T521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 101983 T542 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372300445030/collection1
[junit4:junit4] 2> 101983 T542 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 101984 T542 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 101984 T542 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 101985 T542 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372300445030/collection1/'
[junit4:junit4] 2> 101986 T542 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1372300445030/collection1/lib/README' to classloader
[junit4:junit4] 2> 101987 T542 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1372300445030/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 102028 T542 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 102071 T542 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 102077 T542 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 102086 T542 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 102423 T542 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 102423 T542 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 102424 T542 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 102434 T542 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 102437 T542 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 102461 T542 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 102464 T542 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 102467 T542 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 102468 T542 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 102469 T542 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 102469 T542 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 102470 T542 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 102470 T542 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 102470 T542 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 102470 T542 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372300445030/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1/
[junit4:junit4] 2> 102471 T542 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@135776f4
[junit4:junit4] 2> 102471 T542 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 102471 T542 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1
[junit4:junit4] 2> 102472 T542 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1/index/
[junit4:junit4] 2> 102472 T542 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 102472 T542 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1/index
[junit4:junit4] 2> 102475 T542 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 102476 T542 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 102478 T542 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 102478 T542 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 102479 T542 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 102479 T542 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 102480 T542 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 102480 T542 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 102481 T542 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 102482 T542 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 102482 T542 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 102485 T542 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 102489 T542 oass.SolrIndexSearcher.<init> Opening Searcher@316a4fe6 main
[junit4:junit4] 2> 102491 T542 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 102491 T542 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 102499 T543 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@316a4fe6 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 102502 T542 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 102503 T542 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36509/qs collection:collection1 shard:shard1
[junit4:junit4] 2> 102504 T542 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 102509 T542 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 102511 T542 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 102512 T542 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 102512 T542 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:36509/qs/collection1/
[junit4:junit4] 2> 102512 T542 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 102512 T542 oasc.SyncStrategy.syncToMe http://127.0.0.1:36509/qs/collection1/ has no replicas
[junit4:junit4] 2> 102513 T542 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:36509/qs/collection1/
[junit4:junit4] 2> 102513 T542 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 102736 T522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 102768 T521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 102768 T528 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 102768 T541 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 102818 T542 oasc.ZkController.register We are http://127.0.0.1:36509/qs/collection1/ and leader is http://127.0.0.1:36509/qs/collection1/
[junit4:junit4] 2> 102818 T542 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36509/qs
[junit4:junit4] 2> 102819 T542 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 102819 T542 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 102819 T542 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 102821 T542 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 102822 T499 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 102822 T499 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 102823 T499 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 102941 T499 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 102945 T499 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59442
[junit4:junit4] 2> 102946 T499 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 102946 T499 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 102947 T499 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372300448102
[junit4:junit4] 2> 102947 T499 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372300448102/solr.xml
[junit4:junit4] 2> 102948 T499 oasc.CoreContainer.<init> New CoreContainer 165250600
[junit4:junit4] 2> 102948 T499 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372300448102/'
[junit4:junit4] 2> 102949 T499 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372300448102/'
[junit4:junit4] 2> 103029 T499 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 103030 T499 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 103030 T499 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 103030 T499 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 103031 T499 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 103031 T499 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 103032 T499 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 103032 T499 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 103032 T499 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 103033 T499 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 103040 T499 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 103041 T499 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50348/solr
[junit4:junit4] 2> 103041 T499 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 103042 T499 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 103044 T555 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16e6b2ed name:ZooKeeperConnection Watcher:127.0.0.1:50348 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 103045 T499 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 103047 T499 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 103052 T499 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 103054 T557 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c3d8edc name:ZooKeeperConnection Watcher:127.0.0.1:50348/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 103055 T499 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 103058 T499 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 104062 T499 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59442_qs
[junit4:junit4] 2> 104063 T499 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59442_qs
[junit4:junit4] 2> 104065 T528 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 104065 T521 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 104065 T557 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 104065 T528 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 104065 T541 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 104066 T521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 104066 T541 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 104071 T558 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 104071 T558 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 104273 T522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 104274 T522 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:36509_qs_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:36509_qs",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:36509/qs"}
[junit4:junit4] 2> 104278 T522 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:59442_qs",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59442/qs"}
[junit4:junit4] 2> 104279 T522 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 104279 T522 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 104283 T528 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 104283 T557 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 104283 T541 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 104283 T521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 105072 T558 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372300448102/collection1
[junit4:junit4] 2> 105073 T558 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 105074 T558 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 105074 T558 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 105075 T558 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372300448102/collection1/'
[junit4:junit4] 2> 105076 T558 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1372300448102/collection1/lib/README' to classloader
[junit4:junit4] 2> 105076 T558 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1372300448102/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 105109 T558 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 105149 T558 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 105151 T558 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 105162 T558 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 105620 T558 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 105621 T558 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 105622 T558 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 105630 T558 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 105632 T558 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 105665 T558 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 105671 T558 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 105675 T558 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 105677 T558 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 105677 T558 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 105678 T558 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 105681 T558 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 105681 T558 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 105682 T558 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 105682 T558 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372300448102/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty2/
[junit4:junit4] 2> 105682 T558 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@135776f4
[junit4:junit4] 2> 105683 T558 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 105683 T558 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty2
[junit4:junit4] 2> 105684 T558 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty2/index/
[junit4:junit4] 2> 105684 T558 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 105685 T558 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty2/index
[junit4:junit4] 2> 105699 T558 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 105699 T558 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 105701 T558 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 105702 T558 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 105702 T558 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 105703 T558 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 105703 T558 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 105704 T558 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 105704 T558 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 105705 T558 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 105705 T558 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 105707 T558 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 105710 T558 oass.SolrIndexSearcher.<init> Opening Searcher@1473bede main
[junit4:junit4] 2> 105711 T558 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 105711 T558 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 105715 T559 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1473bede main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 105717 T558 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 105718 T558 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59442/qs collection:collection1 shard:shard1
[junit4:junit4] 2> 105722 T558 oasc.ZkController.register We are http://127.0.0.1:59442/qs/collection1/ and leader is http://127.0.0.1:36509/qs/collection1/
[junit4:junit4] 2> 105722 T558 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59442/qs
[junit4:junit4] 2> 105722 T558 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 105722 T558 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C66 name=collection1 org.apache.solr.core.SolrCore@4b9320b9 url=http://127.0.0.1:59442/qs/collection1 node=127.0.0.1:59442_qs C66_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:59442_qs, base_url=http://127.0.0.1:59442/qs}
[junit4:junit4] 2> 105726 T560 C66 P59442 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 105727 T558 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 105727 T560 C66 P59442 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 105728 T560 C66 P59442 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 105728 T560 C66 P59442 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 105729 T499 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 105729 T499 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 105729 T560 C66 P59442 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 105730 T499 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 105742 T534 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:59442_qs_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 105791 T522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 105792 T522 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:59442_qs_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:59442_qs",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59442/qs"}
[junit4:junit4] 2> 105803 T521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 105804 T541 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 105803 T557 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 105804 T528 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 105833 T499 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 105836 T499 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51661
[junit4:junit4] 2> 105836 T499 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 105837 T499 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 105837 T499 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372300451007
[junit4:junit4] 2> 105838 T499 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372300451007/solr.xml
[junit4:junit4] 2> 105838 T499 oasc.CoreContainer.<init> New CoreContainer 2090428562
[junit4:junit4] 2> 105839 T499 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372300451007/'
[junit4:junit4] 2> 105839 T499 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372300451007/'
[junit4:junit4] 2> 105901 T499 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 105901 T499 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 105901 T499 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 105902 T499 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 105902 T499 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 105902 T499 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 105903 T499 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 105903 T499 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 105904 T499 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 105904 T499 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 105910 T499 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 105911 T499 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50348/solr
[junit4:junit4] 2> 105911 T499 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 105912 T499 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 105914 T572 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a767b0a name:ZooKeeperConnection Watcher:127.0.0.1:50348 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 105914 T499 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 105920 T499 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 105926 T499 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 105927 T574 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b9c51d3 name:ZooKeeperConnection Watcher:127.0.0.1:50348/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 105927 T499 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 105930 T499 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 106742 T534 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:59442_qs_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 106743 T534 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:59442_qs_collection1&state=recovering&nodeName=127.0.0.1:59442_qs&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1003
[junit4:junit4] 2> 106933 T499 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51661_qs
[junit4:junit4] 2> 106934 T499 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51661_qs
[junit4:junit4] 2> 106936 T541 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 106937 T574 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 106937 T541 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 106937 T557 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 106937 T521 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 106937 T557 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 106937 T528 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 106938 T521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 106938 T528 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 106942 T575 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 106942 T575 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 107309 T522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 107310 T522 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:51661_qs",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51661/qs"}
[junit4:junit4] 2> 107310 T522 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 107310 T522 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 107315 T528 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 107316 T557 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 107316 T521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 107316 T541 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 107315 T574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 107943 T575 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372300451007/collection1
[junit4:junit4] 2> 107943 T575 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 107944 T575 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 107944 T575 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 107948 T575 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372300451007/collection1/'
[junit4:junit4] 2> 107949 T575 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1372300451007/collection1/lib/README' to classloader
[junit4:junit4] 2> 107950 T575 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1372300451007/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 108000 T575 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 108048 T575 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 108049 T575 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 108056 T575 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 108415 T575 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 108416 T575 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 108416 T575 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 108423 T575 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 108426 T575 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 108443 T575 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 108454 T575 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 108458 T575 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 108459 T575 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 108459 T575 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 108459 T575 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 108460 T575 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 108460 T575 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 108461 T575 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 108461 T575 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372300451007/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty3/
[junit4:junit4] 2> 108461 T575 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@135776f4
[junit4:junit4] 2> 108462 T575 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 108462 T575 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty3
[junit4:junit4] 2> 108462 T575 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty3/index/
[junit4:junit4] 2> 108462 T575 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 108463 T575 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty3/index
[junit4:junit4] 2> 108466 T575 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 108466 T575 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 108468 T575 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 108468 T575 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 108469 T575 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 108469 T575 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 108470 T575 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 108470 T575 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 108470 T575 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 108471 T575 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 108471 T575 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 108473 T575 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 108476 T575 oass.SolrIndexSearcher.<init> Opening Searcher@7de8b294 main
[junit4:junit4] 2> 108477 T575 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 108477 T575 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 108482 T576 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7de8b294 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 108486 T575 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 108486 T575 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51661/qs collection:collection1 shard:shard1
[junit4:junit4] 2> 108489 T575 oasc.ZkController.register We are http://127.0.0.1:51661/qs/collection1/ and leader is http://127.0.0.1:36509/qs/collection1/
[junit4:junit4] 2> 108489 T575 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51661/qs
[junit4:junit4] 2> 108489 T575 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 108489 T575 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C67 name=collection1 org.apache.solr.core.SolrCore@571d75e9 url=http://127.0.0.1:51661/qs/collection1 node=127.0.0.1:51661_qs C67_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:51661_qs, base_url=http://127.0.0.1:51661/qs}
[junit4:junit4] 2> 108490 T577 C67 P51661 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 108491 T575 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 108491 T577 C67 P51661 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 108492 T577 C67 P51661 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 108492 T577 C67 P51661 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 108492 T499 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 108493 T499 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 108493 T577 C67 P51661 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 108493 T499 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 108501 T536 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:51661_qs_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 108571 T499 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 108573 T499 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52260
[junit4:junit4] 2> 108574 T499 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 108574 T499 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 108575 T499 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372300453772
[junit4:junit4] 2> 108575 T499 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372300453772/solr.xml
[junit4:junit4] 2> 108575 T499 oasc.CoreContainer.<init> New CoreContainer 437684925
[junit4:junit4] 2> 108576 T499 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372300453772/'
[junit4:junit4] 2> 108576 T499 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372300453772/'
[junit4:junit4] 2> 108622 T499 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 108623 T499 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 108623 T499 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 108623 T499 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 108624 T499 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 108624 T499 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 108624 T499 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 108624 T499 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 108625 T499 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 108625 T499 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 108629 T499 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 108629 T499 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50348/solr
[junit4:junit4] 2> 108629 T499 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 108630 T499 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 108632 T589 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5dd3d0ce name:ZooKeeperConnection Watcher:127.0.0.1:50348 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 108632 T499 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 108634 T499 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 108637 T499 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 108639 T591 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4485f287 name:ZooKeeperConnection Watcher:127.0.0.1:50348/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 108639 T499 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 108642 T499 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C68 name=collection1 org.apache.solr.core.SolrCore@4b9320b9 url=http://127.0.0.1:59442/qs/collection1 node=127.0.0.1:59442_qs C68_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:59442_qs, base_url=http://127.0.0.1:59442/qs}
[junit4:junit4] 2> 108745 T560 C68 P59442 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:36509/qs/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 108750 T560 C68 P59442 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 108757 T560 C68 P59442 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59442/qs START replicas=[http://127.0.0.1:36509/qs/collection1/] nUpdates=100
[junit4:junit4] 2> 108761 T560 C68 P59442 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 108762 T560 C68 P59442 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 108762 T560 C68 P59442 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 108762 T560 C68 P59442 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 108763 T560 C68 P59442 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 108763 T560 C68 P59442 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:36509/qs/collection1/. core=collection1
[junit4:junit4] 2> 108764 T560 C68 P59442 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C69 name=collection1 org.apache.solr.core.SolrCore@39763dce url=http://127.0.0.1:36509/qs/collection1 node=127.0.0.1:36509_qs C69_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:36509_qs, base_url=http://127.0.0.1:36509/qs, leader=true}
[junit4:junit4] 2> 108769 T535 C69 P36509 oasc.SolrCore.execute [collection1] webapp=/qs path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 108778 T534 C69 P36509 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 108780 T534 C69 P36509 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 108781 T534 C69 P36509 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 108781 T534 C69 P36509 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 108781 T534 C69 P36509 oass.SolrIndexSearcher.<init> Opening Searcher@8a8b80b realtime
[junit4:junit4] 2> 108782 T534 C69 P36509 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 108782 T534 C69 P36509 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4] 2> 108803 T560 C68 P59442 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 108803 T560 C68 P59442 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4] 2> 108811 T560 C68 P59442 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 108815 T535 C69 P36509 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 108816 T535 C69 P36509 oasc.SolrCore.execute [collection1] webapp=/qs path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 108817 T560 C68 P59442 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 108817 T560 C68 P59442 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 108817 T560 C68 P59442 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 108817 T560 C68 P59442 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 108819 T560 C68 P59442 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 108824 T522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 108825 T522 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:51661_qs_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:51661_qs",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51661/qs"}
[junit4:junit4] 2> 108829 T522 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:59442_qs_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:59442_qs",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59442/qs"}
[junit4:junit4] 2> 108833 T591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 108834 T521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 108834 T541 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 108834 T557 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 108834 T528 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 108833 T574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 109502 T536 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:51661_qs_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 109502 T536 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:51661_qs_collection1&state=recovering&nodeName=127.0.0.1:51661_qs&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 109645 T499 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52260_qs
[junit4:junit4] 2> 109647 T499 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52260_qs
[junit4:junit4] 2> 109649 T591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 109649 T574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 109649 T557 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 109650 T528 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 109650 T528 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 109650 T521 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 109650 T541 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 109650 T557 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 109651 T541 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 109650 T521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 109652 T574 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 109652 T591 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 109658 T593 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 109658 T593 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 110340 T522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 110341 T522 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52260_qs",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52260/qs"}
[junit4:junit4] 2> 110341 T522 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 110341 T522 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 110347 T528 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 110348 T557 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 110348 T521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 110348 T574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 110347 T541 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 110348 T591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 110659 T593 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372300453772/collection1
[junit4:junit4] 2> 110660 T593 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 110660 T593 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 110661 T593 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 110662 T593 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372300453772/collection1/'
[junit4:junit4] 2> 110662 T593 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1372300453772/collection1/lib/README' to classloader
[junit4:junit4] 2> 110662 T593 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1372300453772/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 110698 T593 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 110730 T593 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 110731 T593 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 110738 T593 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 111105 T593 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 111105 T593 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 111107 T593 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 111113 T593 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 111116 T593 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 111138 T593 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 111142 T593 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 111145 T593 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 111146 T593 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 111146 T593 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 111146 T593 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 111147 T593 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 111148 T593 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 111148 T593 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 111148 T593 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372300453772/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty4/
[junit4:junit4] 2> 111149 T593 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@135776f4
[junit4:junit4] 2> 111149 T593 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 111149 T593 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty4
[junit4:junit4] 2> 111150 T593 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty4/index/
[junit4:junit4] 2> 111150 T593 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 111150 T593 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty4/index
[junit4:junit4] 2> 111153 T593 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 111153 T593 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 111156 T593 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 111156 T593 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 111159 T593 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 111159 T593 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 111160 T593 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 111160 T593 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 111160 T593 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 111162 T593 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 111162 T593 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 111164 T593 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 111168 T593 oass.SolrIndexSearcher.<init> Opening Searcher@13577391 main
[junit4:junit4] 2> 111169 T593 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 111170 T593 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 111175 T594 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13577391 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 111178 T593 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 111178 T593 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52260/qs collection:collection1 shard:shard1
[junit4:junit4] 2> 111181 T593 oasc.ZkController.register We are http://127.0.0.1:52260/qs/collection1/ and leader is http://127.0.0.1:36509/qs/collection1/
[junit4:junit4] 2> 111182 T593 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52260/qs
[junit4:junit4] 2> 111182 T593 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 111182 T593 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C70 name=collection1 org.apache.solr.core.SolrCore@4d0ae6c url=http://127.0.0.1:52260/qs/collection1 node=127.0.0.1:52260_qs C70_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:52260_qs, base_url=http://127.0.0.1:52260/qs}
[junit4:junit4] 2> 111184 T595 C70 P52260 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 111184 T593 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 111185 T595 C70 P52260 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 111185 T595 C70 P52260 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 111186 T595 C70 P52260 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 111186 T499 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 111186 T499 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 111187 T499 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 111189 T595 C70 P52260 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 111197 T499 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 111203 T536 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:52260_qs_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 111205 T499 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 111205 T499 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 111206 T499 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C71 name=collection1 org.apache.solr.core.SolrCore@571d75e9 url=http://127.0.0.1:51661/qs/collection1 node=127.0.0.1:51661_qs C71_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:51661_qs, base_url=http://127.0.0.1:51661/qs}
[junit4:junit4] 2> 111504 T577 C71 P51661 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:36509/qs/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 111504 T577 C71 P51661 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:51661/qs START replicas=[http://127.0.0.1:36509/qs/collection1/] nUpdates=100
[junit4:junit4] 2> 111504 T577 C71 P51661 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 111505 T577 C71 P51661 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 111505 T577 C71 P51661 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 111505 T577 C71 P51661 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 111505 T577 C71 P51661 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 111506 T577 C71 P51661 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:36509/qs/collection1/. core=collection1
[junit4:junit4] 2> 111506 T577 C71 P51661 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C72 name=collection1 org.apache.solr.core.SolrCore@39763dce url=http://127.0.0.1:36509/qs/collection1 node=127.0.0.1:36509_qs C72_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:36509_qs, base_url=http://127.0.0.1:36509/qs, leader=true}
[junit4:junit4] 2> 111508 T535 C72 P36509 oasc.SolrCore.execute [collection1] webapp=/qs path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 111512 T537 C72 P36509 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 111512 T537 C72 P36509 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 111513 T537 C72 P36509 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 111513 T537 C72 P36509 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 1
[junit4:junit4] 2> 111514 T577 C71 P51661 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 111514 T577 C71 P51661 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 111518 T535 C72 P36509 oasc.SolrCore.execute [collection1] webapp=/qs path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 111519 T577 C71 P51661 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 111519 T577 C71 P51661 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 111519 T577 C71 P51661 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 111519 T577 C71 P51661 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 111520 T577 C71 P51661 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 111854 T522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 111855 T522 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:52260_qs_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52260_qs",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52260/qs"}
[junit4:junit4] 2> 111858 T522 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:51661_qs_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:51661_qs",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51661/qs"}
[junit4:junit4] 2> 111863 T574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 111863 T557 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 111863 T521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 111863 T528 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 111863 T591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 111863 T541 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 112204 T536 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:52260_qs_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 112204 T536 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:52260_qs_collection1&state=recovering&nodeName=127.0.0.1:52260_qs&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 112208 T499 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 113210 T499 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C70_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:52260_qs, base_url=http://127.0.0.1:52260/qs}
[junit4:junit4] 2> 114206 T595 C70 P52260 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:36509/qs/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 114206 T595 C70 P52260 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52260/qs START replicas=[http://127.0.0.1:36509/qs/collection1/] nUpdates=100
[junit4:junit4] 2> 114207 T595 C70 P52260 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 114207 T595 C70 P52260 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 114207 T595 C70 P52260 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 114207 T595 C70 P52260 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 114207 T595 C70 P52260 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 114208 T595 C70 P52260 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:36509/qs/collection1/. core=collection1
[junit4:junit4] 2> 114208 T595 C70 P52260 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 114209 T535 C72 P36509 oasc.SolrCore.execute [collection1] webapp=/qs path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 114211 T499 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 114215 T536 C72 P36509 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 114216 T536 C72 P36509 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 114216 T536 C72 P36509 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 114217 T536 C72 P36509 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 2
[junit4:junit4] 2> 114218 T595 C70 P52260 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 114218 T595 C70 P52260 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 114221 T535 C72 P36509 oasc.SolrCore.execute [collection1] webapp=/qs path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 114221 T595 C70 P52260 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 114222 T595 C70 P52260 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 114222 T595 C70 P52260 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 114222 T595 C70 P52260 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 114224 T595 C70 P52260 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 114871 T522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 114872 T522 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:52260_qs_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52260_qs",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52260/qs"}
[junit4:junit4] 2> 114876 T591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 114876 T541 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 114876 T557 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 114876 T574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 114876 T521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 114876 T528 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 115213 T499 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 115214 T499 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C73 name=collection1 org.apache.solr.core.SolrCore@4eb2cd7a url=http://127.0.0.1:49822/qs/collection1 node=127.0.0.1:49822_qs C73_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:49822_qs, base_url=http://127.0.0.1:49822/qs, leader=true}
[junit4:junit4] 2> 115221 T515 C73 P49822 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 115223 T515 C73 P49822 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 115224 T515 C73 P49822 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 115224 T515 C73 P49822 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 115225 T515 C73 P49822 oass.SolrIndexSearcher.<init> Opening Searcher@6bf4423a main
[junit4:junit4] 2> 115225 T515 C73 P49822 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 115226 T525 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6bf4423a main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 115227 T515 C73 P49822 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4] 2> ASYNC NEW_CORE C74 name=collection1 org.apache.solr.core.SolrCore@4d0ae6c url=http://127.0.0.1:52260/qs/collection1 node=127.0.0.1:52260_qs C74_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52260_qs, base_url=http://127.0.0.1:52260/qs}
[junit4:junit4] 2> 115249 T585 C74 P52260 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:36509/qs/collection1/, StdNode: http://127.0.0.1:59442/qs/collection1/, StdNode: http://127.0.0.1:51661/qs/collection1/, StdNode: http://127.0.0.1:52260/qs/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> ASYNC NEW_CORE C75 name=collection1 org.apache.solr.core.SolrCore@39763dce url=http://127.0.0.1:36509/qs/collection1 node=127.0.0.1:36509_qs C75_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:36509_qs, base_url=http://127.0.0.1:36509/qs, leader=true}
[junit4:junit4] 2> 115261 T534 C75 P36509 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 115261 T587 C74 P52260 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 115261 T534 C75 P36509 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 115262 T534 C75 P36509 oass.SolrIndexSearcher.<init> Opening Searcher@5974da34 main
[junit4:junit4] 2> 115263 T534 C75 P36509 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 115264 T543 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5974da34 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 115265 T534 C75 P36509 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 115266 T587 C74 P52260 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 115267 T587 C74 P52260 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 115267 T587 C74 P52260 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 115268 T587 C74 P52260 oass.SolrIndexSearcher.<init> Opening Searcher@994856a main
[junit4:junit4] 2> 115268 T587 C74 P52260 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> ASYNC NEW_CORE C76 name=collection1 org.apache.solr.core.SolrCore@4b9320b9 url=http://127.0.0.1:59442/qs/collection1 node=127.0.0.1:59442_qs C76_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:59442_qs, base_url=http://127.0.0.1:59442/qs}
[junit4:junit4] 2> 115269 T552 C76 P59442 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 115269 T594 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@994856a main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 115271 T587 C74 P52260 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 10
[junit4:junit4] 2> ASYNC NEW_CORE C77 name=collection1 org.apache.solr.core.SolrCore@571d75e9 url=http://127.0.0.1:51661/qs/collection1 node=127.0.0.1:51661_qs C77_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:51661_qs, base_url=http://127.0.0.1:51661/qs}
[junit4:junit4] 2> 115272 T570 C77 P51661 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 115273 T552 C76 P59442 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 115274 T552 C76 P59442 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 115274 T552 C76 P59442 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 115275 T552 C76 P59442 oass.SolrIndexSearcher.<init> Opening Searcher@3192675a main
[junit4:junit4] 2> 115275 T552 C76 P59442 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 115275 T570 C77 P51661 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 115276 T570 C77 P51661 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 115276 T559 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3192675a main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 115277 T570 C77 P51661 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 115277 T552 C76 P59442 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 115278 T570 C77 P51661 oass.SolrIndexSearcher.<init> Opening Searcher@5f9e981e main
[junit4:junit4] 2> 115278 T570 C77 P51661 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 115279 T576 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5f9e981e main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 115279 T570 C77 P51661 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 2> 115280 T585 C74 P52260 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 31
[junit4:junit4] 2> 115282 T499 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 115285 T537 C75 P36509 oasc.SolrCore.execute [collection1] webapp=/qs path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 115291 T550 C76 P59442 oasc.SolrCore.execute [collection1] webapp=/qs path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 115294 T567 C77 P51661 oasc.SolrCore.execute [collection1] webapp=/qs path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 115298 T584 C74 P52260 oasc.SolrCore.execute [collection1] webapp=/qs path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> ASYNC NEW_CORE C78 name=collection1 org.apache.solr.core.SolrCore@4eb2cd7a url=http://127.0.0.1:49822/qs/collection1 node=127.0.0.1:49822_qs C78_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:49822_qs, base_url=http://127.0.0.1:49822/qs, leader=true}
[junit4:junit4] 2> 117304 T517 C78 P49822 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1438961329846091776)} 0 3
[junit4:junit4] 2> 117318 T568 C77 P51661 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={update.distrib=FROMLEADER&_version_=-1438961329855528960&update.from=http://127.0.0.1:36509/qs/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438961329855528960)} 0 2
[junit4:junit4] 2> 117319 T587 C74 P52260 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={update.distrib=FROMLEADER&_version_=-1438961329855528960&update.from=http://127.0.0.1:36509/qs/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438961329855528960)} 0 1
[junit4:junit4] 2> 117320 T550 C76 P59442 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={update.distrib=FROMLEADER&_version_=-1438961329855528960&update.from=http://127.0.0.1:36509/qs/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438961329855528960)} 0 1
[junit4:junit4] 2> 117320 T535 C75 P36509 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1438961329855528960)} 0 9
[junit4:junit4] 2> 117321 T567 C77 P51661 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 15
[junit4:junit4] 2> 117328 T515 C78 P49822 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={wt=javabin&version=2} {add=[0 (1438961329868111872)]} 0 4
[junit4:junit4] 2> 117347 T585 C74 P52260 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={distrib.from=http://127.0.0.1:36509/qs/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438961329882791936)]} 0 3
[junit4:junit4] 2> 117353 T551 C76 P59442 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={distrib.from=http://127.0.0.1:36509/qs/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438961329882791936)]} 0 9
[junit4:junit4] 2> 117353 T569 C77 P51661 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={distrib.from=http://127.0.0.1:36509/qs/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438961329882791936)]} 0 8
[junit4:junit4] 2> 117354 T536 C75 P36509 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={distrib.from=http://127.0.0.1:59442/qs/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1438961329882791936)]} 0 16
[junit4:junit4] 2> 117355 T552 C76 P59442 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={wt=javabin&version=2} {add=[0]} 0 25
[junit4:junit4] 2> 117361 T516 C78 P49822 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={wt=javabin&version=2} {add=[1 (1438961329904812032)]} 0 3
[junit4:junit4] 2> 117378 T568 C77 P51661 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={distrib.from=http://127.0.0.1:36509/qs/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438961329915297792)]} 0 3
[junit4:junit4] 2> 117379 T586 C74 P52260 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={distrib.from=http://127.0.0.1:36509/qs/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438961329915297792)]} 0 3
[junit4:junit4] 2> 117381 T550 C76 P59442 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={distrib.from=http://127.0.0.1:36509/qs/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438961329915297792)]} 0 3
[junit4:junit4] 2> 117381 T537 C75 P36509 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={distrib.from=http://127.0.0.1:51661/qs/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1438961329915297792)]} 0 12
[junit4:junit4] 2> 117383 T570 C77 P51661 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={wt=javabin&version=2} {add=[1]} 0 18
[junit4:junit4] 2> 117389 T514 C78 P49822 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={wt=javabin&version=2} {add=[2 (1438961329934172160)]} 0 3
[junit4:junit4] 2> 117397 T535 C75 P36509 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:59442/qs/collection1/]
[junit4:junit4] 2> 117398 T535 C75 P36509 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:59442/qs/collection1/ against:[http://127.0.0.1:59442/qs/collection1/] result:true
[junit4:junit4] 2> 117398 T535 C75 P36509 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:51661/qs/collection1/ against:[http://127.0.0.1:59442/qs/collection1/] result:false
[junit4:junit4] 2> 117398 T535 C75 P36509 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:52260/qs/collection1/ against:[http://127.0.0.1:59442/qs/collection1/] result:false
[junit4:junit4] 2> 117409 T567 C77 P51661 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={distrib.from=http://127.0.0.1:36509/qs/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:59442/qs/collection1/&wt=javabin&version=2} {add=[2 (1438961329946755072)]} 0 3
[junit4:junit4] 2> 117409 T584 C74 P52260 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={distrib.from=http://127.0.0.1:36509/qs/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:59442/qs/collection1/&wt=javabin&version=2} {add=[2 (1438961329946755072)]} 0 3
[junit4:junit4] 2> 117410 T535 C75 P36509 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={distrib.from=http://127.0.0.1:59442/qs/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:59442/qs/collection1/&wt=javabin&version=2} {add=[2 (1438961329946755072)]} 0 13
[junit4:junit4] 2> 117411 T553 C76 P59442 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={test.distrib.skip.servers=http://127.0.0.1:59442/qs/collection1/&wt=javabin&version=2} {add=[2]} 0 18
[junit4:junit4] 2> 117417 T517 C78 P49822 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={wt=javabin&version=2} {add=[3 (1438961329962483712)]} 0 3
[junit4:junit4] 2> 117425 T534 C75 P36509 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:59442/qs/collection1/]
[junit4:junit4] 2> 117426 T534 C75 P36509 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:59442/qs/collection1/ against:[http://127.0.0.1:59442/qs/collection1/] result:true
[junit4:junit4] 2> 117427 T534 C75 P36509 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:51661/qs/collection1/ against:[http://127.0.0.1:59442/qs/collection1/] result:false
[junit4:junit4] 2> 117427 T534 C75 P36509 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:52260/qs/collection1/ against:[http://127.0.0.1:59442/qs/collection1/] result:false
[junit4:junit4] 2> 117435 T569 C77 P51661 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={distrib.from=http://127.0.0.1:36509/qs/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:59442/qs/collection1/&wt=javabin&version=2} {add=[3 (1438961329976115200)]} 0 2
[junit4:junit4] 2> 117435 T585 C74 P52260 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={distrib.from=http://127.0.0.1:36509/qs/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:59442/qs/collection1/&wt=javabin&version=2} {add=[3 (1438961329976115200)]} 0 2
[junit4:junit4] 2> 117437 T534 C75 P36509 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={distrib.from=http://127.0.0.1:52260/qs/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:59442/qs/collection1/&wt=javabin&version=2} {add=[3 (1438961329976115200)]} 0 12
[junit4:junit4] 2> 117438 T587 C74 P52260 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={test.distrib.skip.servers=http://127.0.0.1:59442/qs/collection1/&test.distrib.skip.servers=http://127.0.0.1:51661/qs/collection1/&wt=javabin&version=2} {add=[3]} 0 17
[junit4:junit4] 2> 117442 T515 C78 P49822 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 117497 T515 C78 P49822 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/control/data/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 117498 T515 C78 P49822 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 117501 T515 C78 P49822 oass.SolrIndexSearcher.<init> Opening Searcher@4b152bf main
[junit4:junit4] 2> 117502 T515 C78 P49822 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 117502 T525 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4b152bf main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 117503 T515 C78 P49822 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 61
[junit4:junit4] 2> 117507 T586 C74 P52260 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:36509/qs/collection1/, StdNode: http://127.0.0.1:59442/qs/collection1/, StdNode: http://127.0.0.1:51661/qs/collection1/, StdNode: http://127.0.0.1:52260/qs/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 117510 T551 C76 P59442 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 117510 T584 C74 P52260 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 117510 T568 C77 P51661 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 117512 T536 C75 P36509 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 117537 T551 C76 P59442 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 117538 T551 C76 P59442 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 117540 T551 C76 P59442 oass.SolrIndexSearcher.<init> Opening Searcher@4d316b82 main
[junit4:junit4] 2> 117541 T551 C76 P59442 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 117541 T559 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4d316b82 main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c2)}
[junit4:junit4] 2> 117542 T551 C76 P59442 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 32
[junit4:junit4] 2> 117567 T568 C77 P51661 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty3/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 117568 T568 C77 P51661 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 117567 T536 C75 P36509 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 117569 T536 C75 P36509 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 117570 T568 C77 P51661 oass.SolrIndexSearcher.<init> Opening Searcher@52258bc7 main
[junit4:junit4] 2> 117570 T536 C75 P36509 oass.SolrIndexSearcher.<init> Opening Searcher@71b1a2e8 main
[junit4:junit4] 2> 117571 T568 C77 P51661 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 117571 T536 C75 P36509 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 117571 T543 oasc.JmxMonitoredMap.put WARN Failed to register info bean: searcher javax.management.InstanceAlreadyExistsException: solr/collection1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:513)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4] 2> at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:298)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1846)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1723)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2>
[junit4:junit4] 2> 117572 T576 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@52258bc7 main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 117573 T543 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@71b1a2e8 main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 117573 T568 C77 P51661 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 63
[junit4:junit4] 2> 117573 T536 C75 P36509 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 62
[junit4:junit4] 2> 117578 T584 C74 P52260 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 117578 T584 C74 P52260 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 117581 T584 C74 P52260 oass.SolrIndexSearcher.<init> Opening Searcher@765b86d7 main
[junit4:junit4] 2> 117582 T584 C74 P52260 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 117582 T594 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@765b86d7 main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 117583 T584 C74 P52260 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 73
[junit4:junit4] 2> 117584 T586 C74 P52260 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 78
[junit4:junit4] 2> 117584 T499 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 117585 T499 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 117586 T499 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 117589 T537 C75 P36509 oasc.SolrCore.execute [collection1] webapp=/qs path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 117592 T552 C76 P59442 oasc.SolrCore.execute [collection1] webapp=/qs path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0
[junit4:junit4] 2> 117596 T570 C77 P51661 oasc.SolrCore.execute [collection1] webapp=/qs path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 117600 T585 C74 P52260 oasc.SolrCore.execute [collection1] webapp=/qs path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 117603 T499 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 117614 T569 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4] 2> 117615 T569 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 117622 T535 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4] 2> 117622 T535 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:36509/qs/collection1/
[junit4:junit4] 2> 117622 T535 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:36509/qs START replicas=[http://127.0.0.1:59442/qs/collection1/, http://127.0.0.1:51661/qs/collection1/, http://127.0.0.1:52260/qs/collection1/] nUpdates=100
[junit4:junit4] 2> 117628 T570 C77 P51661 oasc.SolrCore.execute [collection1] webapp=/qs path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=2
[junit4:junit4] 2> 117629 T587 C74 P52260 oasc.SolrCore.execute [collection1] webapp=/qs path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 117629 T550 C76 P59442 oasc.SolrCore.execute [collection1] webapp=/qs path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=2
[junit4:junit4] 2> 117630 T535 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:36509/qs Received 5 versions from 127.0.0.1:52260/qs/collection1/
[junit4:junit4] 2> 117630 T535 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:36509/qs Our versions are newer. ourLowThreshold=1438961329855528960 otherHigh=1438961329946755072
[junit4:junit4] 2> 117631 T535 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:36509/qs Received 3 versions from 127.0.0.1:59442/qs/collection1/
[junit4:junit4] 2> 117631 T535 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:36509/qs Our versions are newer. ourLowThreshold=1438961329855528960 otherHigh=1438961329915297792
[junit4:junit4] 2> 117632 T535 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:36509/qs Received 5 versions from 127.0.0.1:51661/qs/collection1/
[junit4:junit4] 2> 117632 T535 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:36509/qs Our versions are newer. ourLowThreshold=1438961329855528960 otherHigh=1438961329946755072
[junit4:junit4] 2> 117632 T535 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:36509/qs DONE. sync succeeded
[junit4:junit4] 2> 117633 T535 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 117633 T535 oasc.SyncStrategy.syncToMe http://127.0.0.1:36509/qs/collection1/: try and ask http://127.0.0.1:59442/qs/collection1/ to sync
[junit4:junit4] 2> 117634 T535 oasc.SyncStrategy.syncToMe http://127.0.0.1:36509/qs/collection1/: try and ask http://127.0.0.1:51661/qs/collection1/ to sync
[junit4:junit4] 2> 117634 T535 oasc.SyncStrategy.syncToMe http://127.0.0.1:36509/qs/collection1/: try and ask http://127.0.0.1:52260/qs/collection1/ to sync
[junit4:junit4] 2> 117637 T553 C76 P59442 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59442/qs START replicas=[http://127.0.0.1:36509/qs/collection1/] nUpdates=100
[junit4:junit4] 2> 117646 T586 C74 P52260 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52260/qs START replicas=[http://127.0.0.1:36509/qs/collection1/] nUpdates=100
[junit4:junit4] 2> 117646 T568 C77 P51661 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:51661/qs START replicas=[http://127.0.0.1:36509/qs/collection1/] nUpdates=100
[junit4:junit4] 2> 117648 T534 C75 P36509 oasc.SolrCore.execute [collection1] webapp=/qs path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 117651 T553 C76 P59442 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59442/qs Received 5 versions from 127.0.0.1:36509/qs/collection1/
[junit4:junit4] 2> 117651 T553 C76 P59442 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:59442/qs Requesting updates from 127.0.0.1:36509/qs/collection1/n=2 versions=[1438961329976115200, 1438961329946755072]
[junit4:junit4] 2> 117652 T537 C75 P36509 oasc.SolrCore.execute [collection1] webapp=/qs path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 117652 T536 C75 P36509 oasc.SolrCore.execute [collection1] webapp=/qs path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 117653 T568 C77 P51661 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:51661/qs Received 5 versions from 127.0.0.1:36509/qs/collection1/
[junit4:junit4] 2> 117660 T568 C77 P51661 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:51661/qs Our versions are newer. ourLowThreshold=1438961329855528960 otherHigh=1438961329946755072
[junit4:junit4] 2> 117660 T568 C77 P51661 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:51661/qs DONE. sync succeeded
[junit4:junit4] 2> 117660 T586 C74 P52260 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:52260/qs Received 5 versions from 127.0.0.1:36509/qs/collection1/
[junit4:junit4] 2> 117660 T568 C77 P51661 oasc.SolrCore.execute [collection1] webapp=/qs path=/get params={sync=http://127.0.0.1:36509/qs/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=14
[junit4:junit4] 2> 117661 T586 C74 P52260 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:52260/qs Our versions are newer. ourLowThreshold=1438961329855528960 otherHigh=1438961329946755072
[junit4:junit4] 2> 117661 T534 C75 P36509 oasc.SolrCore.execute [collection1] webapp=/qs path=/get params={distrib=false&getUpdates=1438961329976115200,1438961329946755072&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 117661 T586 C74 P52260 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52260/qs DONE. sync succeeded
[junit4:junit4] 2> 117662 T535 oasc.SyncStrategy.syncToMe http://127.0.0.1:36509/qs/collection1/: sync completed with http://127.0.0.1:51661/qs/collection1/
[junit4:junit4] 2> 117662 T586 C74 P52260 oasc.SolrCore.execute [collection1] webapp=/qs path=/get params={sync=http://127.0.0.1:36509/qs/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=17
[junit4:junit4] 2> 117666 T535 oasc.SyncStrategy.syncToMe http://127.0.0.1:36509/qs/collection1/: sync completed with http://127.0.0.1:52260/qs/collection1/
[junit4:junit4] 2> 117670 T553 C76 P59442 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1438961329946755072), 3 (1438961329976115200)]} 0 7
[junit4:junit4] 2> 117670 T553 C76 P59442 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59442/qs DONE. sync succeeded
[junit4:junit4] 2> 117671 T553 C76 P59442 oasc.SolrCore.execute [collection1] webapp=/qs path=/get params={sync=http://127.0.0.1:36509/qs/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=34
[junit4:junit4] 2> 117672 T535 oasc.SyncStrategy.syncToMe http://127.0.0.1:36509/qs/collection1/: sync completed with http://127.0.0.1:59442/qs/collection1/
[junit4:junit4] 2> 117672 T535 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=50
[junit4:junit4] 2> 117673 T569 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=61
[junit4:junit4] 2> 117674 T499 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 117674 T499 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 117675 T499 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 117676 T499 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 117679 T516 C78 P49822 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 117680 T516 C78 P49822 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 117681 T516 C78 P49822 oass.SolrIndexSearcher.<init> Opening Searcher@2fe6f0b4 main
[junit4:junit4] 2> 117681 T516 C78 P49822 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 117682 T525 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2fe6f0b4 main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 117683 T516 C78 P49822 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4] 2> 117686 T551 C76 P59442 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:36509/qs/collection1/, StdNode: http://127.0.0.1:59442/qs/collection1/, StdNode: http://127.0.0.1:51661/qs/collection1/, StdNode: http://127.0.0.1:52260/qs/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 117690 T536 C75 P36509 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 117690 T536 C75 P36509 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 117691 T536 C75 P36509 oass.SolrIndexSearcher.<init> Opening Searcher@1d51caa9 main
[junit4:junit4] 2> 117691 T567 C77 P51661 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 117692 T567 C77 P51661 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 117692 T567 C77 P51661 oass.SolrIndexSearcher.<init> Opening Searcher@67fcafd0 main
[junit4:junit4] 2> 117693 T567 C77 P51661 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 117690 T587 C74 P52260 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 117693 T543 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d51caa9 main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 117695 T553 C76 P59442 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 117692 T536 C75 P36509 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 117694 T587 C74 P52260 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 117698 T536 C75 P36509 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 117733 T576 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@67fcafd0 main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> ASYNC NEW_CORE C79 name=collection1 org.apache.solr.core.SolrCore@4d0ae6c url=http://127.0.0.1:52260/qs/collection1 node=127.0.0.1:52260_qs C79_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52260_qs, base_url=http://127.0.0.1:52260/qs}
[junit4:junit4] 2> 117698 T587 C79 P52260 oass.SolrIndexSearcher.<init> Opening Searcher@774dde17 main
[junit4:junit4] 2> ASYNC NEW_CORE C80 name=collection1 org.apache.solr.core.SolrCore@571d75e9 url=http://127.0.0.1:51661/qs/collection1 node=127.0.0.1:51661_qs C80_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:51661_qs, base_url=http://127.0.0.1:51661/qs}
[junit4:junit4] 2> 117734 T567 C80 P51661 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 43
[junit4:junit4] 2> 117735 T587 C79 P52260 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 117737 T594 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@774dde17 main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 117738 T587 C79 P52260 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 48
[junit4:junit4] 2> ASYNC NEW_CORE C81 name=collection1 org.apache.solr.core.SolrCore@4b9320b9 url=http://127.0.0.1:59442/qs/collection1 node=127.0.0.1:59442_qs C81_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:59442_qs, base_url=http://127.0.0.1:59442/qs}
[junit4:junit4] 2> 117817 T553 C81 P59442 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty2/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> 117818 T553 C81 P59442 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 117821 T553 C81 P59442 oass.SolrIndexSearcher.<init> Opening Searcher@6e2637c3 main
[junit4:junit4] 2> 117822 T553 C81 P59442 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 117823 T559 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6e2637c3 main{StandardDirectoryReader(segments_3:5:nrt _0(4.4):c2 _1(4.4):c2)}
[junit4:junit4] 2> 117824 T553 C81 P59442 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 129
[junit4:junit4] 2> 117825 T551 C81 P59442 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 139
[junit4:junit4] 2> 117826 T499 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C82 name=collection1 org.apache.solr.core.SolrCore@39763dce url=http://127.0.0.1:36509/qs/collection1 node=127.0.0.1:36509_qs C82_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:36509_qs, base_url=http://127.0.0.1:36509/qs, leader=true}
[junit4:junit4] 2> 117830 T537 C82 P36509 oasc.SolrCore.execute [collection1] webapp=/qs path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 117834 T550 C81 P59442 oasc.SolrCore.execute [collection1] webapp=/qs path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 117837 T569 C80 P51661 oasc.SolrCore.execute [collection1] webapp=/qs path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> 117840 T586 C79 P52260 oasc.SolrCore.execute [collection1] webapp=/qs path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 119841 T499 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> check const of shard1
[junit4:junit4] 2> client0
[junit4:junit4] 2> PROPS:127.0.0.1:36509_qs_collection1:{"shard":"shard1","state":"active","core":"collection1","collection":"collection1","node_name":"127.0.0.1:36509_qs","base_url":"http://127.0.0.1:36509/qs","leader":"true"}
[junit4:junit4] 2> 119845 T535 C82 P36509 oasc.SolrCore.execute [collection1] webapp=/qs path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client1
[junit4:junit4] 2> PROPS:127.0.0.1:59442_qs_collection1:{"shard":"shard1","state":"active","core":"collection1","collection":"collection1","node_name":"127.0.0.1:59442_qs","base_url":"http://127.0.0.1:59442/qs"}
[junit4:junit4] 2> 119848 T552 C81 P59442 oasc.SolrCore.execute [collection1] webapp=/qs path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client2
[junit4:junit4] 2> PROPS:127.0.0.1:51661_qs_collection1:{"shard":"shard1","state":"active","core":"collection1","collection":"collection1","node_name":"127.0.0.1:51661_qs","base_url":"http://127.0.0.1:51661/qs"}
[junit4:junit4] 2> 119852 T568 C80 P51661 oasc.SolrCore.execute [collection1] webapp=/qs path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client3
[junit4:junit4] 2> PROPS:127.0.0.1:52260_qs_collection1:{"shard":"shard1","state":"active","core":"collection1","collection":"collection1","node_name":"127.0.0.1:52260_qs","base_url":"http://127.0.0.1:52260/qs"}
[junit4:junit4] 2> 119855 T584 C79 P52260 oasc.SolrCore.execute [collection1] webapp=/qs path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> 119857 T534 C82 P36509 oasc.SolrCore.execute [collection1] webapp=/qs path=/select params={q=*:*&wt=javabin&version=2} hits=4 status=0 QTime=0
[junit4:junit4] 2> ASYNC NEW_CORE C83 name=collection1 org.apache.solr.core.SolrCore@4eb2cd7a url=http://127.0.0.1:49822/qs/collection1 node=127.0.0.1:49822_qs C83_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:49822_qs, base_url=http://127.0.0.1:49822/qs, leader=true}
[junit4:junit4] 2> 119865 T516 C83 P49822 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={wt=javabin&version=2} {add=[4 (1438961332528349184)]} 0 4
[junit4:junit4] 2> 119871 T536 C82 P36509 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:51661/qs/collection1/]
[junit4:junit4] 2> 119872 T536 C82 P36509 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:59442/qs/collection1/ against:[http://127.0.0.1:51661/qs/collection1/] result:false
[junit4:junit4] 2> 119872 T536 C82 P36509 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:51661/qs/collection1/ against:[http://127.0.0.1:51661/qs/collection1/] result:true
[junit4:junit4] 2> 119872 T536 C82 P36509 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:52260/qs/collection1/ against:[http://127.0.0.1:51661/qs/collection1/] result:false
[junit4:junit4] 2> 119881 T585 C79 P52260 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={distrib.from=http://127.0.0.1:36509/qs/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:51661/qs/collection1/&wt=javabin&version=2} {add=[4 (1438961332540932096)]} 0 3
[junit4:junit4] 2> 119881 T550 C81 P59442 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={distrib.from=http://127.0.0.1:36509/qs/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:51661/qs/collection1/&wt=javabin&version=2} {add=[4 (1438961332540932096)]} 0 3
[junit4:junit4] 2> 119882 T536 C82 P36509 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={distrib.from=http://127.0.0.1:51661/qs/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:51661/qs/collection1/&wt=javabin&version=2} {add=[4 (1438961332540932096)]} 0 11
[junit4:junit4] 2> 119883 T569 C80 P51661 oasup.LogUpdateProcessor.finish [collection1] webapp=/qs path=/update params={test.distrib.skip.servers=http://127.0.0.1:51661/qs/collection1/&wt=javabin&version=2} {add=[4]} 0 15
[junit4:junit4] 2> 119883 T499 oasc.ChaosMonkey.monkeyLog monkey: kill shard! 36509
[junit4:junit4] 2> 119905 T499 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1045900807
[junit4:junit4] 2> 120886 T522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 120887 T522 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:36509_qs_collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:36509_qs",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:36509/qs"}
[junit4:junit4] 2> 120890 T591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 120890 T574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 120890 T521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 120890 T528 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 120890 T557 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 120890 T541 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 121909 T499 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 121910 T499 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 121911 T499 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@39763dce
[junit4:junit4] 2> 121922 T499 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=5,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 121922 T499 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 121923 T499 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 121923 T499 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> C82_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=down, core=collection1, collection=collection1, node_name=127.0.0.1:36509_qs, base_url=http://127.0.0.1:36509/qs, leader=true}
[junit4:junit4] 2> 122097 T499 C82 P36509 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> 122098 T499 C82 P36509 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 122101 T499 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 122102 T499 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 122102 T499 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1/index;done=false>>]
[junit4:junit4] 2> 122102 T499 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1/index
[junit4:junit4] 2> 122103 T499 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1;done=false>>]
[junit4:junit4] 2> 122103 T499 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372300441530/jetty1
[junit4:junit4] 2> 122105 T591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 122106 T574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 122106 T521 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 122106 T521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 122107 T528 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 122107 T528 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 122107 T499 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/qs,null}
[junit4:junit4] 2> 122108 T591 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 122108 T557 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 122109 T574 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 122110 T557 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 122111 T557 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 122111 T557 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 122111 T557 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59442/qs/collection1/
[junit4:junit4] 2> 122111 T557 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59442/qs START replicas=[http://127.0.0.1:36509/qs/collection1/, http://127.0.0.1:51661/qs/collection1/, http://127.0.0.1:52260/qs/collection1/] nUpdates=100
[junit4:junit4] 2> 122114 T557 oasu.PeerSync.handleResponse WARN PeerSync: core=collection1 url=http://127.0.0.1:59442/qs couldn't connect to http://127.0.0.1:36509/qs/collection1/, counting as success
[junit4:junit4] 2> 122116 T567 C80 P51661 oasc.SolrCore.execute [collection1] webapp=/qs path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 122116 T585 C79 P52260 oasc.SolrCore.execute [collection1] webapp=/qs path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 122117 T557 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59442/qs Received 6 versions from 127.0.0.1:52260/qs/collection1/
[junit4:junit4] 2> 122117 T557 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59442/qs Our versions are newer. ourLowThreshold=1438961329882791936 otherHigh=1438961329976115200
[junit4:junit4] 2> 122117 T557 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59442/qs Received 5 versions from 127.0.0.1:51661/qs/collection1/
[junit4:junit4] 2> 122117 T557 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59442/qs Our versions are newer. ourLowThreshold=1438961329882791936 otherHigh=1438961329946755072
[junit4:junit4] 2> 122118 T557 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59442/qs DONE. sync succeeded
[junit4:junit4] 2> 122118 T557 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 122118 T557 oasc.SyncStrategy.syncToMe http://127.0.0.1:59442/qs/collection1/: try and ask http://127.0.0.1:36509/qs/collection1/ to sync
[junit4:junit4] 2> 122118 T557 oasc.SyncStrategy.syncToMe http://127.0.0.1:59442/qs/collection1/: try and ask http://127.0.0.1:51661/qs/collection1/ to sync
[junit4:junit4] 2> 122118 T557 oasc.SyncStrategy.syncToMe http://127.0.0.1:59442/qs/collection1/: try and ask http://127.0.0.1:52260/qs/collection1/ to sync
[junit4:junit4] 2> 122120 T557 oasc.SolrException.log ERROR Sync request error: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:36509/qs/collection1
[junit4:junit4] 2> 122120 T557 oasc.SyncStrategy.syncToMe http://127.0.0.1:59442/qs/collection1/: Sync failed - asking replica (http://127.0.0.1:36509/qs/collection1/) to recover.
[junit4:junit4] 2> 122121 T627 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 122122 T568 C80 P51661 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:51661/qs START replicas=[http://127.0.0.1:59442/qs/collection1/] nUpdates=100
[junit4:junit4] 2> 122124 T586 C79 P52260 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52260/qs START replicas=[http://127.0.0.1:59442/qs/collection1/] nUpdates=100
[junit4:junit4] 2> 122127 T627 oasc.SolrException.log ERROR http://127.0.0.1:59442/qs/collection1/: Could not tell a replica to recover:org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:36509/qs
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:429)
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4] 2> at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:298)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:36509 refused
[junit4:junit4] 2> at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
[junit4:junit4] 2> at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4:junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4:junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4:junit4] 2> ... 5 more
[junit4:junit4] 2> Caused by: java.net.ConnectException: Connection refused
[junit4:junit4] 2> at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4:junit4] 2> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
[junit4:junit4] 2> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
[junit4:junit4] 2> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
[junit4:junit4] 2> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
[junit4:junit4] 2> at java.net.Socket.connect(Socket.java:581)
[junit4:junit4] 2> at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4:junit4] 2> at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4:junit4] 2> ... 12 more
[junit4:junit4] 2>
[junit4:junit4] 2> 122128 T552 C81 P59442 oasc.SolrCore.execute [collection1] webapp=/qs path=/get params
[...truncated too long message...]
"replicas":{
[junit4:junit4] 1> "127.0.0.1:36509_qs_collection1":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:36509_qs",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:36509/qs"},
[junit4:junit4] 1> "127.0.0.1:59442_qs_collection1":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "state":"down",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:59442_qs",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:59442/qs"},
[junit4:junit4] 1> "127.0.0.1:51661_qs_collection1":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:51661_qs",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:51661/qs",
[junit4:junit4] 1> "leader":"true"},
[junit4:junit4] 1> "127.0.0.1:52260_qs_collection1":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:52260_qs",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:52260/qs"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"127.0.0.1:49822_qs_collection1":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "node_name":"127.0.0.1:49822_qs",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:49822/qs",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:36509_qs (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:52260_qs (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:49822_qs (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:51661_qs (0)
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/collections (2)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89935081736306700-127.0.0.1:52260_qs_collection1-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89935081736306702-127.0.0.1:36509_qs_collection1-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89935081736306698-127.0.0.1:51661_qs_collection1-n_0000000002 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (1)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:51661_qs",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:51661/qs"}
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89935081736306691-127.0.0.1:49822_qs_collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:49822_qs",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:49822/qs"}
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/election (4)
[junit4:junit4] 1> /solr/overseer_elect/election/89935081736306698-127.0.0.1:51661_qs-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89935081736306702-127.0.0.1:36509_qs-n_0000000005 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89935081736306700-127.0.0.1:52260_qs-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89935081736306691-127.0.0.1:49822_qs-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89935081736306691-127.0.0.1:49822_qs-n_0000000000"}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=9671151141C19F1F -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=is -Dtests.timezone=Asia/Kuwait -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE 105s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:36509/qs/collection1lastClient and got 246 from http://127.0.0.1:51661/qs/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([9671151141C19F1F:17979B09369EFF23]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 201370 T499 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 105130 T498 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4] 2> NOTE: test params are: codec=Asserting, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=is, timezone=Asia/Kuwait
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.8.0-ea (64-bit)/cpus=8,threads=3,free=63961624,total=133873664
[junit4:junit4] 2> NOTE: All tests run in this JVM: [ResponseLogComponentTest, RegexBoostProcessorTest, SimpleFacetsTest, OverseerTest, TestSolrIndexConfig, AutoCommitTest, TestRemoteStreaming, TestWriterPerf, TestLFUCache, WordBreakSolrSpellCheckerTest, SpellingQueryConverterTest, DirectSolrConnectionTest, TestSweetSpotSimilarityFactory, TestValueSourceCache, SystemInfoHandlerTest, TestSolrCoreProperties, TestSolrDeletionPolicy2, SyncSliceTest]
[junit4:junit4] Completed on J0 in 105.73s, 1 test, 1 failure <<< FAILURES!
[...truncated 744 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1246: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:890: There were test failures: 299 suites, 1289 tests, 1 failure, 16 ignored (7 assumptions)
Total time: 38 minutes 5 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-ea-b94 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_25) - Build # 6257
- Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6257/
Java: 32bit/jdk1.7.0_25 -server -XX:+UseConcMarkSweepGC
1 tests failed.
FAILED: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:38377/hpqtn/wh/collection1lastClient and got 250 from http://127.0.0.1:52456/hpqtn/wh/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:38377/hpqtn/wh/collection1lastClient and got 250 from http://127.0.0.1:52456/hpqtn/wh/collection1
at __randomizedtesting.SeedInfo.seed([4B7778AE2F424EAE:CA91F6B6581D2E92]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9724 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4] 2> 771473 T2795 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /hpqtn/wh
[junit4:junit4] 2> 771477 T2795 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1372306017327
[junit4:junit4] 2> 771478 T2795 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 771478 T2796 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 771578 T2795 oasc.ZkTestServer.run start zk server on port:35337
[junit4:junit4] 2> 771579 T2795 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 773072 T2799 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 1491ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4:junit4] 2> 773073 T2802 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b9760e name:ZooKeeperConnection Watcher:127.0.0.1:35337 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 773074 T2795 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 773074 T2795 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 773082 T2795 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 773083 T2804 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e3a94b name:ZooKeeperConnection Watcher:127.0.0.1:35337/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 773084 T2795 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 773084 T2795 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 773086 T2795 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 773088 T2795 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 773090 T2795 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 773092 T2795 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 773092 T2795 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 773095 T2795 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 773096 T2795 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 773099 T2795 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 773099 T2795 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 773101 T2795 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 773101 T2795 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 773102 T2795 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 773103 T2795 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 773105 T2795 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 773105 T2795 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 773107 T2795 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 773107 T2795 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 773109 T2795 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 773110 T2795 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 773111 T2795 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 773112 T2795 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 773201 T2795 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 773213 T2795 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56990
[junit4:junit4] 2> 773214 T2795 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 773214 T2795 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 773214 T2795 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372306018963
[junit4:junit4] 2> 773215 T2795 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372306018963/solr.xml
[junit4:junit4] 2> 773215 T2795 oasc.CoreContainer.<init> New CoreContainer 13313266
[junit4:junit4] 2> 773215 T2795 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372306018963/'
[junit4:junit4] 2> 773216 T2795 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372306018963/'
[junit4:junit4] 2> 773279 T2795 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 773279 T2795 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 773280 T2795 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 773280 T2795 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 773281 T2795 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 773281 T2795 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 773281 T2795 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 773282 T2795 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 773282 T2795 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 773282 T2795 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 773286 T2795 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 773286 T2795 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35337/solr
[junit4:junit4] 2> 773287 T2795 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 773292 T2795 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 773292 T2815 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e02cc name:ZooKeeperConnection Watcher:127.0.0.1:35337 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 773293 T2795 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 773294 T2795 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 773312 T2795 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 773313 T2817 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@223a5 name:ZooKeeperConnection Watcher:127.0.0.1:35337/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 773313 T2795 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 773314 T2795 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 773317 T2795 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 773320 T2795 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 773321 T2795 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56990_hpqtn%2Fwh
[junit4:junit4] 2> 773322 T2795 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56990_hpqtn%2Fwh
[junit4:junit4] 2> 773324 T2795 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 773329 T2795 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 773330 T2795 oasc.Overseer.start Overseer (id=89935447151673347-127.0.0.1:56990_hpqtn%2Fwh-n_0000000000) starting
[junit4:junit4] 2> 773332 T2795 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 773335 T2819 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 773336 T2795 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 773337 T2795 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 773339 T2795 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 773341 T2818 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 773343 T2820 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 773343 T2820 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 774848 T2818 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 774849 T2818 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56990_hpqtn%2Fwh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56990/hpqtn/wh"}
[junit4:junit4] 2> 774849 T2818 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 774849 T2818 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 774854 T2817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 775344 T2820 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372306018963/collection1
[junit4:junit4] 2> 775345 T2820 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 775345 T2820 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 775346 T2820 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 775347 T2820 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372306018963/collection1/'
[junit4:junit4] 2> 775347 T2820 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1372306018963/collection1/lib/README' to classloader
[junit4:junit4] 2> 775347 T2820 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1372306018963/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 775373 T2820 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 775409 T2820 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 775410 T2820 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 775413 T2820 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 775923 T2820 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 775924 T2820 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 775924 T2820 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 775930 T2820 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 775933 T2820 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 775963 T2820 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 775978 T2820 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 775981 T2820 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 775983 T2820 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 775983 T2820 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 775983 T2820 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 775985 T2820 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 775985 T2820 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 775985 T2820 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 775985 T2820 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372306018963/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372306017326/control/data/
[junit4:junit4] 2> 775986 T2820 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@dd0e8a
[junit4:junit4] 2> 775986 T2820 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 775987 T2820 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/control/data
[junit4:junit4] 2> 775987 T2820 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372306017326/control/data/index/
[junit4:junit4] 2> 775987 T2820 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372306017326/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 775988 T2820 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/control/data/index
[junit4:junit4] 2> 776002 T2820 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 776002 T2820 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 776004 T2820 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 776005 T2820 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 776006 T2820 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 776006 T2820 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 776007 T2820 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 776007 T2820 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 776007 T2820 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 776011 T2820 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 776011 T2820 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 776015 T2820 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 776018 T2820 oass.SolrIndexSearcher.<init> Opening Searcher@c866d9 main
[junit4:junit4] 2> 776019 T2820 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 776019 T2820 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 776023 T2820 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 776024 T2820 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56990/hpqtn/wh collection:control_collection shard:shard1
[junit4:junit4] 2> 776024 T2820 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 776024 T2821 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c866d9 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 776030 T2820 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 776032 T2820 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 776032 T2820 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 776032 T2820 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56990/hpqtn/wh/collection1/
[junit4:junit4] 2> 776033 T2820 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 776033 T2820 oasc.SyncStrategy.syncToMe http://127.0.0.1:56990/hpqtn/wh/collection1/ has no replicas
[junit4:junit4] 2> 776033 T2820 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56990/hpqtn/wh/collection1/
[junit4:junit4] 2> 776033 T2820 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 776356 T2818 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 776362 T2817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 776389 T2820 oasc.ZkController.register We are http://127.0.0.1:56990/hpqtn/wh/collection1/ and leader is http://127.0.0.1:56990/hpqtn/wh/collection1/
[junit4:junit4] 2> 776389 T2820 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56990/hpqtn/wh
[junit4:junit4] 2> 776389 T2820 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 776389 T2820 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 776390 T2820 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 776392 T2820 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 776393 T2795 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 776393 T2795 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 776393 T2795 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 776396 T2795 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 776428 T2795 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 776428 T2824 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3934c7 name:ZooKeeperConnection Watcher:127.0.0.1:35337/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 776429 T2795 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 776430 T2795 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 776432 T2795 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 776497 T2795 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 776499 T2795 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38377
[junit4:junit4] 2> 776500 T2795 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 776500 T2795 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 776500 T2795 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372306022281
[junit4:junit4] 2> 776500 T2795 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372306022281/solr.xml
[junit4:junit4] 2> 776501 T2795 oasc.CoreContainer.<init> New CoreContainer 9766679
[junit4:junit4] 2> 776501 T2795 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372306022281/'
[junit4:junit4] 2> 776501 T2795 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372306022281/'
[junit4:junit4] 2> 776540 T2795 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 776540 T2795 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 776541 T2795 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 776541 T2795 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 776541 T2795 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 776541 T2795 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 776542 T2795 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 776542 T2795 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 776542 T2795 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 776543 T2795 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 776545 T2795 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 776546 T2795 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35337/solr
[junit4:junit4] 2> 776546 T2795 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 776547 T2795 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 776548 T2835 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ba5378 name:ZooKeeperConnection Watcher:127.0.0.1:35337 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 776548 T2795 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 776550 T2795 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 776553 T2795 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 776554 T2837 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ba070c name:ZooKeeperConnection Watcher:127.0.0.1:35337/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 776554 T2795 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 776558 T2795 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 777562 T2795 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38377_hpqtn%2Fwh
[junit4:junit4] 2> 777563 T2795 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38377_hpqtn%2Fwh
[junit4:junit4] 2> 777574 T2817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 777575 T2837 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 777575 T2824 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 777576 T2817 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 777579 T2838 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 777580 T2838 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 777867 T2818 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 777868 T2818 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:56990_hpqtn%2Fwh_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56990_hpqtn%2Fwh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56990/hpqtn/wh"}
[junit4:junit4] 2> 777872 T2818 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:38377_hpqtn%2Fwh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38377/hpqtn/wh"}
[junit4:junit4] 2> 777872 T2818 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 777872 T2818 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 777874 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 777874 T2817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 777875 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 778581 T2838 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372306022281/collection1
[junit4:junit4] 2> 778581 T2838 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 778581 T2838 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 778582 T2838 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 778582 T2838 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372306022281/collection1/'
[junit4:junit4] 2> 778583 T2838 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1372306022281/collection1/lib/README' to classloader
[junit4:junit4] 2> 778583 T2838 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1372306022281/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 778618 T2838 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 778646 T2838 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 778647 T2838 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 778650 T2838 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 778967 T2838 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 778967 T2838 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 778967 T2838 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 778972 T2838 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 778975 T2838 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 778984 T2838 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 778987 T2838 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 778989 T2838 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 778990 T2838 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 778990 T2838 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 778990 T2838 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 778991 T2838 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 778991 T2838 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 778991 T2838 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 778992 T2838 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372306022281/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1/
[junit4:junit4] 2> 778992 T2838 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@dd0e8a
[junit4:junit4] 2> 778992 T2838 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 778993 T2838 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1
[junit4:junit4] 2> 778993 T2838 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1/index/
[junit4:junit4] 2> 778993 T2838 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 778993 T2838 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1/index
[junit4:junit4] 2> 778996 T2838 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 778996 T2838 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 778998 T2838 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 778998 T2838 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 778998 T2838 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 778999 T2838 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 778999 T2838 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 778999 T2838 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 779000 T2838 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 779000 T2838 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 779000 T2838 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 779002 T2838 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 779005 T2838 oass.SolrIndexSearcher.<init> Opening Searcher@e1fc5a main
[junit4:junit4] 2> 779005 T2838 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 779005 T2838 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 779009 T2839 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e1fc5a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 779010 T2838 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 779010 T2838 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38377/hpqtn/wh collection:collection1 shard:shard1
[junit4:junit4] 2> 779011 T2838 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 779016 T2838 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 779017 T2838 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 779018 T2838 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 779018 T2838 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38377/hpqtn/wh/collection1/
[junit4:junit4] 2> 779018 T2838 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 779018 T2838 oasc.SyncStrategy.syncToMe http://127.0.0.1:38377/hpqtn/wh/collection1/ has no replicas
[junit4:junit4] 2> 779018 T2838 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38377/hpqtn/wh/collection1/
[junit4:junit4] 2> 779018 T2838 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 779378 T2818 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 779383 T2817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 779383 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 779383 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 779423 T2838 oasc.ZkController.register We are http://127.0.0.1:38377/hpqtn/wh/collection1/ and leader is http://127.0.0.1:38377/hpqtn/wh/collection1/
[junit4:junit4] 2> 779424 T2838 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38377/hpqtn/wh
[junit4:junit4] 2> 779424 T2838 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 779424 T2838 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 779424 T2838 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 779425 T2838 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 779426 T2795 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 779426 T2795 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 779426 T2795 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 779500 T2795 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 779502 T2795 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34506
[junit4:junit4] 2> 779502 T2795 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 779503 T2795 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 779503 T2795 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372306025278
[junit4:junit4] 2> 779503 T2795 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372306025278/solr.xml
[junit4:junit4] 2> 779504 T2795 oasc.CoreContainer.<init> New CoreContainer 2705303
[junit4:junit4] 2> 779504 T2795 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372306025278/'
[junit4:junit4] 2> 779504 T2795 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372306025278/'
[junit4:junit4] 2> 779544 T2795 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 779545 T2795 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 779545 T2795 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 779545 T2795 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 779546 T2795 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 779546 T2795 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 779546 T2795 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 779546 T2795 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 779547 T2795 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 779547 T2795 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 779550 T2795 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 779550 T2795 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35337/solr
[junit4:junit4] 2> 779550 T2795 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 779551 T2795 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 779552 T2851 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f2ea6a name:ZooKeeperConnection Watcher:127.0.0.1:35337 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 779552 T2795 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 779554 T2795 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 779557 T2795 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 779558 T2853 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@126e3ec name:ZooKeeperConnection Watcher:127.0.0.1:35337/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 779558 T2795 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 779560 T2795 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 780563 T2795 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34506_hpqtn%2Fwh
[junit4:junit4] 2> 780564 T2795 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34506_hpqtn%2Fwh
[junit4:junit4] 2> 780565 T2817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 780565 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 780566 T2837 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 780567 T2853 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 780567 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 780567 T2817 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 780567 T2824 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 780571 T2854 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 780571 T2854 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 780887 T2818 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 780888 T2818 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:38377_hpqtn%2Fwh_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:38377_hpqtn%2Fwh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38377/hpqtn/wh"}
[junit4:junit4] 2> 780891 T2818 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:34506_hpqtn%2Fwh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:34506/hpqtn/wh"}
[junit4:junit4] 2> 780891 T2818 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 780891 T2818 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 780894 T2817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 780894 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 780894 T2853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 780894 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 781572 T2854 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372306025278/collection1
[junit4:junit4] 2> 781572 T2854 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 781573 T2854 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 781573 T2854 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 781574 T2854 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372306025278/collection1/'
[junit4:junit4] 2> 781575 T2854 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1372306025278/collection1/lib/README' to classloader
[junit4:junit4] 2> 781575 T2854 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1372306025278/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 781599 T2854 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 781632 T2854 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 781633 T2854 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 781637 T2854 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 781924 T2854 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 781925 T2854 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 781925 T2854 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 781930 T2854 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 781932 T2854 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 781944 T2854 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 781947 T2854 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 781949 T2854 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 781950 T2854 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 781950 T2854 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 781950 T2854 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 781951 T2854 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 781951 T2854 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 781951 T2854 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 781952 T2854 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372306025278/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty2/
[junit4:junit4] 2> 781952 T2854 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@dd0e8a
[junit4:junit4] 2> 781952 T2854 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 781952 T2854 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty2
[junit4:junit4] 2> 781953 T2854 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty2/index/
[junit4:junit4] 2> 781953 T2854 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 781953 T2854 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty2/index
[junit4:junit4] 2> 781955 T2854 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 781956 T2854 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 781958 T2854 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 781958 T2854 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 781959 T2854 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 781959 T2854 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 781960 T2854 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 781960 T2854 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 781961 T2854 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 781961 T2854 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 781962 T2854 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 781965 T2854 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 781968 T2854 oass.SolrIndexSearcher.<init> Opening Searcher@bf2f0a main
[junit4:junit4] 2> 781969 T2854 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 781969 T2854 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 781974 T2855 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@bf2f0a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 781975 T2854 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 781975 T2854 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:34506/hpqtn/wh collection:collection1 shard:shard1
[junit4:junit4] 2> 781978 T2854 oasc.ZkController.register We are http://127.0.0.1:34506/hpqtn/wh/collection1/ and leader is http://127.0.0.1:38377/hpqtn/wh/collection1/
[junit4:junit4] 2> 781978 T2854 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:34506/hpqtn/wh
[junit4:junit4] 2> 781978 T2854 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 781978 T2854 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C3901 name=collection1 org.apache.solr.core.SolrCore@61dfb8 url=http://127.0.0.1:34506/hpqtn/wh/collection1 node=127.0.0.1:34506_hpqtn%2Fwh C3901_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:34506_hpqtn%2Fwh, base_url=http://127.0.0.1:34506/hpqtn/wh}
[junit4:junit4] 2> 781979 T2856 C3901 P34506 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 781979 T2856 C3901 P34506 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 781979 T2854 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 781979 T2856 C3901 P34506 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 781980 T2856 C3901 P34506 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 781980 T2795 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 781981 T2795 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 781980 T2856 C3901 P34506 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 781981 T2795 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 781984 T2833 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:34506_hpqtn%2Fwh_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 782064 T2795 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 782066 T2795 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52456
[junit4:junit4] 2> 782066 T2795 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 782067 T2795 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 782067 T2795 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372306027833
[junit4:junit4] 2> 782067 T2795 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372306027833/solr.xml
[junit4:junit4] 2> 782068 T2795 oasc.CoreContainer.<init> New CoreContainer 17697646
[junit4:junit4] 2> 782068 T2795 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372306027833/'
[junit4:junit4] 2> 782069 T2795 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372306027833/'
[junit4:junit4] 2> 782119 T2795 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 782120 T2795 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 782120 T2795 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 782120 T2795 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 782121 T2795 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 782121 T2795 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 782121 T2795 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 782121 T2795 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 782122 T2795 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 782122 T2795 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 782125 T2795 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 782125 T2795 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35337/solr
[junit4:junit4] 2> 782125 T2795 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 782126 T2795 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 782127 T2868 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1924188 name:ZooKeeperConnection Watcher:127.0.0.1:35337 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 782127 T2795 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 782129 T2795 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 782132 T2795 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 782133 T2870 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a5a9c name:ZooKeeperConnection Watcher:127.0.0.1:35337/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 782133 T2795 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 782136 T2795 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 782398 T2818 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 782398 T2818 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:34506_hpqtn%2Fwh_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:34506_hpqtn%2Fwh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:34506/hpqtn/wh"}
[junit4:junit4] 2> 782401 T2817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 782401 T2870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 782401 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 782401 T2853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 782401 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 782985 T2833 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:34506_hpqtn%2Fwh_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 782985 T2833 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:34506_hpqtn%252Fwh_collection1&state=recovering&nodeName=127.0.0.1:34506_hpqtn%252Fwh&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 783139 T2795 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52456_hpqtn%2Fwh
[junit4:junit4] 2> 783140 T2795 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52456_hpqtn%2Fwh
[junit4:junit4] 2> 783142 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 783142 T2817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 783142 T2837 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 783142 T2853 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 783142 T2870 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 783142 T2853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 783142 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 783143 T2870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 783143 T2824 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 783144 T2817 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 783148 T2871 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 783148 T2871 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 783904 T2818 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 783905 T2818 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52456_hpqtn%2Fwh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52456/hpqtn/wh"}
[junit4:junit4] 2> 783905 T2818 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 783905 T2818 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 783908 T2817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 783909 T2853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 783908 T2870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 783908 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 783908 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 784149 T2871 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372306027833/collection1
[junit4:junit4] 2> 784150 T2871 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 784150 T2871 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 784151 T2871 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 784152 T2871 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372306027833/collection1/'
[junit4:junit4] 2> 784152 T2871 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1372306027833/collection1/lib/README' to classloader
[junit4:junit4] 2> 784152 T2871 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1372306027833/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 784175 T2871 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 784202 T2871 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 784204 T2871 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 784207 T2871 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 784504 T2871 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 784504 T2871 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 784505 T2871 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 784510 T2871 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 784512 T2871 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 784521 T2871 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 784524 T2871 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 784526 T2871 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 784527 T2871 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 784527 T2871 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 784528 T2871 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 784528 T2871 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 784529 T2871 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 784529 T2871 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 784529 T2871 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372306027833/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty3/
[junit4:junit4] 2> 784529 T2871 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@dd0e8a
[junit4:junit4] 2> 784529 T2871 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 784530 T2871 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty3
[junit4:junit4] 2> 784530 T2871 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty3/index/
[junit4:junit4] 2> 784530 T2871 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 784530 T2871 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty3/index
[junit4:junit4] 2> 784533 T2871 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 784533 T2871 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 784535 T2871 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 784535 T2871 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 784536 T2871 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 784536 T2871 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 784536 T2871 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 784537 T2871 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 784537 T2871 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 784537 T2871 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 784538 T2871 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 784540 T2871 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 784542 T2871 oass.SolrIndexSearcher.<init> Opening Searcher@9a5df2 main
[junit4:junit4] 2> 784542 T2871 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 784543 T2871 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 784546 T2872 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9a5df2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 784548 T2871 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 784548 T2871 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52456/hpqtn/wh collection:collection1 shard:shard1
[junit4:junit4] 2> 784551 T2871 oasc.ZkController.register We are http://127.0.0.1:52456/hpqtn/wh/collection1/ and leader is http://127.0.0.1:38377/hpqtn/wh/collection1/
[junit4:junit4] 2> 784551 T2871 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52456/hpqtn/wh
[junit4:junit4] 2> 784551 T2871 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 784551 T2871 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C3902 name=collection1 org.apache.solr.core.SolrCore@71e6e6 url=http://127.0.0.1:52456/hpqtn/wh/collection1 node=127.0.0.1:52456_hpqtn%2Fwh C3902_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:52456_hpqtn%2Fwh, base_url=http://127.0.0.1:52456/hpqtn/wh}
[junit4:junit4] 2> 784551 T2873 C3902 P52456 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 784552 T2871 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 784552 T2873 C3902 P52456 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 784552 T2873 C3902 P52456 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 784552 T2873 C3902 P52456 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 784553 T2795 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 784553 T2795 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 784553 T2873 C3902 P52456 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 784554 T2795 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 784557 T2831 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:52456_hpqtn%2Fwh_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 784617 T2795 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 784619 T2795 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:32859
[junit4:junit4] 2> 784619 T2795 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 784619 T2795 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 784620 T2795 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372306030405
[junit4:junit4] 2> 784620 T2795 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372306030405/solr.xml
[junit4:junit4] 2> 784620 T2795 oasc.CoreContainer.<init> New CoreContainer 22696850
[junit4:junit4] 2> 784621 T2795 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372306030405/'
[junit4:junit4] 2> 784621 T2795 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372306030405/'
[junit4:junit4] 2> 784659 T2795 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 784659 T2795 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 784659 T2795 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 784660 T2795 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 784660 T2795 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 784660 T2795 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 784660 T2795 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 784661 T2795 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 784661 T2795 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 784661 T2795 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 784664 T2795 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 784664 T2795 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35337/solr
[junit4:junit4] 2> 784665 T2795 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 784665 T2795 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 784667 T2885 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13e3f2f name:ZooKeeperConnection Watcher:127.0.0.1:35337 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 784667 T2795 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 784669 T2795 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 784671 T2795 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 784672 T2887 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@618a80 name:ZooKeeperConnection Watcher:127.0.0.1:35337/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 784673 T2795 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 784676 T2795 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C3903 name=collection1 org.apache.solr.core.SolrCore@61dfb8 url=http://127.0.0.1:34506/hpqtn/wh/collection1 node=127.0.0.1:34506_hpqtn%2Fwh C3903_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:34506_hpqtn%2Fwh, base_url=http://127.0.0.1:34506/hpqtn/wh}
[junit4:junit4] 2> 784986 T2856 C3903 P34506 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:38377/hpqtn/wh/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 784986 T2856 C3903 P34506 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:34506/hpqtn/wh START replicas=[http://127.0.0.1:38377/hpqtn/wh/collection1/] nUpdates=100
[junit4:junit4] 2> 784987 T2856 C3903 P34506 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 784987 T2856 C3903 P34506 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 784987 T2856 C3903 P34506 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 784987 T2856 C3903 P34506 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 784987 T2856 C3903 P34506 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 784987 T2856 C3903 P34506 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:38377/hpqtn/wh/collection1/. core=collection1
[junit4:junit4] 2> 784987 T2856 C3903 P34506 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C3904 name=collection1 org.apache.solr.core.SolrCore@937626 url=http://127.0.0.1:38377/hpqtn/wh/collection1 node=127.0.0.1:38377_hpqtn%2Fwh C3904_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:38377_hpqtn%2Fwh, base_url=http://127.0.0.1:38377/hpqtn/wh, leader=true}
[junit4:junit4] 2> 784991 T2830 C3904 P38377 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 784993 T2833 C3904 P38377 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 784995 T2833 C3904 P38377 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 784996 T2833 C3904 P38377 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 784996 T2833 C3904 P38377 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 784997 T2833 C3904 P38377 oass.SolrIndexSearcher.<init> Opening Searcher@1c457f9 realtime
[junit4:junit4] 2> 784997 T2833 C3904 P38377 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 784997 T2833 C3904 P38377 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4] 2> 784998 T2856 C3903 P34506 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 784998 T2856 C3903 P34506 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 784999 T2832 C3904 P38377 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 785000 T2832 C3904 P38377 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 785000 T2856 C3903 P34506 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 785000 T2856 C3903 P34506 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 785001 T2856 C3903 P34506 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 785001 T2856 C3903 P34506 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 785002 T2856 C3903 P34506 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 785413 T2818 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 785413 T2818 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:52456_hpqtn%2Fwh_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52456_hpqtn%2Fwh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52456/hpqtn/wh"}
[junit4:junit4] 2> 785417 T2818 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:34506_hpqtn%2Fwh_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:34506_hpqtn%2Fwh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:34506/hpqtn/wh"}
[junit4:junit4] 2> 785420 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 785420 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 785420 T2853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 785420 T2817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 785420 T2887 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 785420 T2870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 785557 T2831 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:52456_hpqtn%2Fwh_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 785558 T2831 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:52456_hpqtn%252Fwh_collection1&state=recovering&nodeName=127.0.0.1:52456_hpqtn%252Fwh&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 785684 T2795 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32859_hpqtn%2Fwh
[junit4:junit4] 2> 785685 T2795 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:32859_hpqtn%2Fwh
[junit4:junit4] 2> 785687 T2817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 785687 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 785687 T2887 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 785687 T2853 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 785688 T2853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 785687 T2837 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 785687 T2870 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 785688 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 785689 T2817 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 785688 T2824 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 785688 T2887 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 785689 T2870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 785694 T2889 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 785694 T2889 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 786925 T2818 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 786925 T2818 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:32859_hpqtn%2Fwh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:32859/hpqtn/wh"}
[junit4:junit4] 2> 786925 T2818 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 786926 T2818 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 786929 T2887 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 786929 T2870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 786929 T2853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 786929 T2817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 786929 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 786929 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> ASYNC NEW_CORE C3905 name=collection1 org.apache.solr.core.SolrCore@71e6e6 url=http://127.0.0.1:52456/hpqtn/wh/collection1 node=127.0.0.1:52456_hpqtn%2Fwh C3905_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:52456_hpqtn%2Fwh, base_url=http://127.0.0.1:52456/hpqtn/wh}
[junit4:junit4] 2> 787559 T2873 C3905 P52456 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:38377/hpqtn/wh/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 787559 T2873 C3905 P52456 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52456/hpqtn/wh START replicas=[http://127.0.0.1:38377/hpqtn/wh/collection1/] nUpdates=100
[junit4:junit4] 2> 787559 T2873 C3905 P52456 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 787560 T2873 C3905 P52456 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 787560 T2873 C3905 P52456 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 787560 T2873 C3905 P52456 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 787560 T2873 C3905 P52456 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 787560 T2873 C3905 P52456 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:38377/hpqtn/wh/collection1/. core=collection1
[junit4:junit4] 2> 787560 T2873 C3905 P52456 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 787561 T2830 C3904 P38377 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 787563 T2833 C3904 P38377 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 787563 T2833 C3904 P38377 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 787564 T2833 C3904 P38377 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 787564 T2833 C3904 P38377 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 1
[junit4:junit4] 2> 787565 T2873 C3905 P52456 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 787565 T2873 C3905 P52456 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 787567 T2830 C3904 P38377 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 787567 T2873 C3905 P52456 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 787567 T2873 C3905 P52456 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 787567 T2873 C3905 P52456 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 787567 T2873 C3905 P52456 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 787568 T2873 C3905 P52456 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 787695 T2889 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372306030405/collection1
[junit4:junit4] 2> 787695 T2889 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 787696 T2889 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 787696 T2889 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 787697 T2889 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372306030405/collection1/'
[junit4:junit4] 2> 787697 T2889 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1372306030405/collection1/lib/README' to classloader
[junit4:junit4] 2> 787698 T2889 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1372306030405/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 787729 T2889 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 787756 T2889 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 787757 T2889 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 787760 T2889 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 788087 T2889 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 788087 T2889 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 788088 T2889 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 788093 T2889 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 788095 T2889 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 788103 T2889 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 788105 T2889 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 788107 T2889 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 788108 T2889 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 788108 T2889 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 788108 T2889 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 788109 T2889 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 788109 T2889 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 788109 T2889 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 788110 T2889 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372306030405/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty4/
[junit4:junit4] 2> 788110 T2889 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@dd0e8a
[junit4:junit4] 2> 788110 T2889 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 788110 T2889 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty4
[junit4:junit4] 2> 788111 T2889 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty4/index/
[junit4:junit4] 2> 788111 T2889 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 788111 T2889 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty4/index
[junit4:junit4] 2> 788114 T2889 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 788114 T2889 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 788115 T2889 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 788116 T2889 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 788116 T2889 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 788117 T2889 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 788117 T2889 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 788117 T2889 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 788118 T2889 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 788119 T2889 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 788119 T2889 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 788122 T2889 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 788125 T2889 oass.SolrIndexSearcher.<init> Opening Searcher@569238 main
[junit4:junit4] 2> 788125 T2889 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 788125 T2889 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 788129 T2891 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@569238 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 788131 T2889 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 788131 T2889 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:32859/hpqtn/wh collection:collection1 shard:shard1
[junit4:junit4] 2> 788133 T2889 oasc.ZkController.register We are http://127.0.0.1:32859/hpqtn/wh/collection1/ and leader is http://127.0.0.1:38377/hpqtn/wh/collection1/
[junit4:junit4] 2> 788133 T2889 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:32859/hpqtn/wh
[junit4:junit4] 2> 788133 T2889 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 788133 T2889 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C3906 name=collection1 org.apache.solr.core.SolrCore@10579f2 url=http://127.0.0.1:32859/hpqtn/wh/collection1 node=127.0.0.1:32859_hpqtn%2Fwh C3906_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:32859_hpqtn%2Fwh, base_url=http://127.0.0.1:32859/hpqtn/wh}
[junit4:junit4] 2> 788134 T2892 C3906 P32859 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 788134 T2889 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 788134 T2892 C3906 P32859 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 788134 T2892 C3906 P32859 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 788134 T2892 C3906 P32859 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 788135 T2795 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 788135 T2795 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 788135 T2892 C3906 P32859 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 788136 T2795 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 788139 T2795 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 788139 T2831 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:32859_hpqtn%2Fwh_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 788140 T2795 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 788140 T2795 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 788141 T2795 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 788433 T2818 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 788433 T2818 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:52456_hpqtn%2Fwh_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52456_hpqtn%2Fwh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52456/hpqtn/wh"}
[junit4:junit4] 2> 788435 T2818 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:32859_hpqtn%2Fwh_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:32859_hpqtn%2Fwh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:32859/hpqtn/wh"}
[junit4:junit4] 2> 788438 T2887 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 788438 T2853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 788438 T2870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 788438 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 788438 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 788438 T2817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 789140 T2831 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:32859_hpqtn%2Fwh_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 789140 T2831 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:32859_hpqtn%252Fwh_collection1&state=recovering&nodeName=127.0.0.1:32859_hpqtn%252Fwh&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 789142 T2795 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 790143 T2795 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C3906_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:32859_hpqtn%2Fwh, base_url=http://127.0.0.1:32859/hpqtn/wh}
[junit4:junit4] 2> 791141 T2892 C3906 P32859 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:38377/hpqtn/wh/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 791141 T2892 C3906 P32859 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:32859/hpqtn/wh START replicas=[http://127.0.0.1:38377/hpqtn/wh/collection1/] nUpdates=100
[junit4:junit4] 2> 791142 T2892 C3906 P32859 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 791142 T2892 C3906 P32859 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 791142 T2892 C3906 P32859 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 791142 T2892 C3906 P32859 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 791142 T2892 C3906 P32859 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 791142 T2892 C3906 P32859 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:38377/hpqtn/wh/collection1/. core=collection1
[junit4:junit4] 2> 791142 T2892 C3906 P32859 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C3907 name=collection1 org.apache.solr.core.SolrCore@937626 url=http://127.0.0.1:38377/hpqtn/wh/collection1 node=127.0.0.1:38377_hpqtn%2Fwh C3907_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:38377_hpqtn%2Fwh, base_url=http://127.0.0.1:38377/hpqtn/wh, leader=true}
[junit4:junit4] 2> 791143 T2831 C3907 P38377 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 791145 T2795 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 791145 T2830 C3907 P38377 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 791145 T2830 C3907 P38377 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 791146 T2830 C3907 P38377 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 791146 T2830 C3907 P38377 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 1
[junit4:junit4] 2> 791147 T2892 C3906 P32859 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 791147 T2892 C3906 P32859 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 791148 T2831 C3907 P38377 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 791149 T2892 C3906 P32859 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 791149 T2892 C3906 P32859 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 791149 T2892 C3906 P32859 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 791149 T2892 C3906 P32859 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 791150 T2892 C3906 P32859 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 791444 T2818 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 791445 T2818 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:32859_hpqtn%2Fwh_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:32859_hpqtn%2Fwh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:32859/hpqtn/wh"}
[junit4:junit4] 2> 791448 T2887 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 791448 T2853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 791448 T2817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 791448 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 791448 T2870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 791448 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 792146 T2795 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 792147 T2795 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C3908 name=collection1 org.apache.solr.core.SolrCore@185dd28 url=http://127.0.0.1:56990/hpqtn/wh/collection1 node=127.0.0.1:56990_hpqtn%2Fwh C3908_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:56990_hpqtn%2Fwh, base_url=http://127.0.0.1:56990/hpqtn/wh, leader=true}
[junit4:junit4] 2> 792152 T2813 C3908 P56990 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 792154 T2813 C3908 P56990 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 792155 T2813 C3908 P56990 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 792155 T2813 C3908 P56990 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 792156 T2813 C3908 P56990 oass.SolrIndexSearcher.<init> Opening Searcher@ecad12 main
[junit4:junit4] 2> 792156 T2813 C3908 P56990 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 792157 T2821 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ecad12 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 792157 T2813 C3908 P56990 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 792160 T2833 C3907 P38377 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:38377/hpqtn/wh/collection1/, StdNode: http://127.0.0.1:34506/hpqtn/wh/collection1/, StdNode: http://127.0.0.1:52456/hpqtn/wh/collection1/, StdNode: http://127.0.0.1:32859/hpqtn/wh/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 792162 T2830 C3907 P38377 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 792162 T2830 C3907 P38377 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 792163 T2830 C3907 P38377 oass.SolrIndexSearcher.<init> Opening Searcher@1963ee7 main
[junit4:junit4] 2> 792163 T2830 C3907 P38377 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 792164 T2839 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1963ee7 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 792165 T2830 C3907 P38377 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4] 2> ASYNC NEW_CORE C3909 name=collection1 org.apache.solr.core.SolrCore@71e6e6 url=http://127.0.0.1:52456/hpqtn/wh/collection1 node=127.0.0.1:52456_hpqtn%2Fwh C3909_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52456_hpqtn%2Fwh, base_url=http://127.0.0.1:52456/hpqtn/wh}
[junit4:junit4] 2> 792165 T2863 C3909 P52456 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> C3906_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:32859_hpqtn%2Fwh, base_url=http://127.0.0.1:32859/hpqtn/wh}
[junit4:junit4] 2> 792166 T2880 C3906 P32859 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C3910 name=collection1 org.apache.solr.core.SolrCore@61dfb8 url=http://127.0.0.1:34506/hpqtn/wh/collection1 node=127.0.0.1:34506_hpqtn%2Fwh C3910_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:34506_hpqtn%2Fwh, base_url=http://127.0.0.1:34506/hpqtn/wh}
[junit4:junit4] 2> 792165 T2848 C3910 P34506 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 792167 T2863 C3909 P52456 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 792167 T2880 C3906 P32859 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 792168 T2848 C3910 P34506 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 792168 T2863 C3909 P52456 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 792168 T2848 C3910 P34506 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 792168 T2880 C3906 P32859 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 792169 T2848 C3910 P34506 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 792169 T2863 C3909 P52456 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 792170 T2848 C3910 P34506 oass.SolrIndexSearcher.<init> Opening Searcher@428a18 main
[junit4:junit4] 2> 792169 T2880 C3906 P32859 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 792170 T2848 C3910 P34506 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 792170 T2863 C3909 P52456 oass.SolrIndexSearcher.<init> Opening Searcher@1f68406 main
[junit4:junit4] 2> 792171 T2855 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@428a18 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 792171 T2880 C3906 P32859 oass.SolrIndexSearcher.<init> Opening Searcher@1d0597 main
[junit4:junit4] 2> 792172 T2848 C3910 P34506 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 2> 792171 T2863 C3909 P52456 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 792172 T2872 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f68406 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 792172 T2880 C3906 P32859 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 792173 T2863 C3909 P52456 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 792172 T2891 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d0597 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4] 2> 792173 T2880 C3906 P32859 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 792174 T2833 C3907 P38377 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 15
[junit4:junit4] 2> 792174 T2795 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 792176 T2832 C3907 P38377 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 792177 T2847 C3910 P34506 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 792178 T2865 C3909 P52456 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 792179 T2882 C3906 P32859 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 794182 T2812 C3908 P56990 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1438967178231545856)} 0 1
[junit4:junit4] 2> 794187 T2880 C3906 P32859 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={update.distrib=FROMLEADER&_version_=-1438967178234691584&update.from=http://127.0.0.1:38377/hpqtn/wh/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438967178234691584)} 0 0
[junit4:junit4] 2> 794187 T2846 C3910 P34506 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={update.distrib=FROMLEADER&_version_=-1438967178234691584&update.from=http://127.0.0.1:38377/hpqtn/wh/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438967178234691584)} 0 0
[junit4:junit4] 2> 794187 T2864 C3909 P52456 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={update.distrib=FROMLEADER&_version_=-1438967178234691584&update.from=http://127.0.0.1:38377/hpqtn/wh/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438967178234691584)} 0 0
[junit4:junit4] 2> 794188 T2833 C3907 P38377 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1438967178234691584)} 0 3
[junit4:junit4] 2> 794189 T2865 C3909 P52456 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 5
[junit4:junit4] 2> 794192 T2813 C3908 P56990 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={wt=javabin&version=2} {add=[0 (1438967178239934464)]} 0 2
[junit4:junit4] 2> 794197 T2863 C3909 P52456 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={distrib.from=http://127.0.0.1:38377/hpqtn/wh/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438967178244128768)]} 0 1
[junit4:junit4] 2> 794197 T2881 C3906 P32859 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={distrib.from=http://127.0.0.1:38377/hpqtn/wh/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438967178244128768)]} 0 1
[junit4:junit4] 2> 794197 T2847 C3910 P34506 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={distrib.from=http://127.0.0.1:38377/hpqtn/wh/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438967178244128768)]} 0 1
[junit4:junit4] 2> 794198 T2832 C3907 P38377 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={distrib.from=http://127.0.0.1:32859/hpqtn/wh/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1438967178244128768)]} 0 4
[junit4:junit4] 2> 794198 T2883 C3906 P32859 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={wt=javabin&version=2} {add=[0]} 0 5
[junit4:junit4] 2> 794201 T2810 C3908 P56990 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={wt=javabin&version=2} {add=[1 (1438967178250420224)]} 0 1
[junit4:junit4] 2> 794205 T2866 C3909 P52456 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={distrib.from=http://127.0.0.1:38377/hpqtn/wh/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438967178253565952)]} 0 0
[junit4:junit4] 2> 794205 T2848 C3910 P34506 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={distrib.from=http://127.0.0.1:38377/hpqtn/wh/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438967178253565952)]} 0 0
[junit4:junit4] 2> 794205 T2882 C3906 P32859 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={distrib.from=http://127.0.0.1:38377/hpqtn/wh/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438967178253565952)]} 0 0
[junit4:junit4] 2> 794206 T2831 C3907 P38377 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={distrib.from=http://127.0.0.1:34506/hpqtn/wh/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1438967178253565952)]} 0 3
[junit4:junit4] 2> 794207 T2849 C3910 P34506 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={wt=javabin&version=2} {add=[1]} 0 5
[junit4:junit4] 2> 794209 T2811 C3908 P56990 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={wt=javabin&version=2} {add=[2 (1438967178258808832)]} 0 1
[junit4:junit4] 2> 794211 T2830 C3907 P38377 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:34506/hpqtn/wh/collection1/]
[junit4:junit4] 2> 794211 T2830 C3907 P38377 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:34506/hpqtn/wh/collection1/ against:[http://127.0.0.1:34506/hpqtn/wh/collection1/] result:true
[junit4:junit4] 2> 794211 T2830 C3907 P38377 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:52456/hpqtn/wh/collection1/ against:[http://127.0.0.1:34506/hpqtn/wh/collection1/] result:false
[junit4:junit4] 2> 794212 T2830 C3907 P38377 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:32859/hpqtn/wh/collection1/ against:[http://127.0.0.1:34506/hpqtn/wh/collection1/] result:false
[junit4:junit4] 2> 794214 T2880 C3906 P32859 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={distrib.from=http://127.0.0.1:38377/hpqtn/wh/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:34506/hpqtn/wh/collection1/&wt=javabin&version=2} {add=[2 (1438967178263003136)]} 0 0
[junit4:junit4] 2> 794214 T2864 C3909 P52456 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={distrib.from=http://127.0.0.1:38377/hpqtn/wh/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:34506/hpqtn/wh/collection1/&wt=javabin&version=2} {add=[2 (1438967178263003136)]} 0 0
[junit4:junit4] 2> 794215 T2830 C3907 P38377 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={test.distrib.skip.servers=http://127.0.0.1:34506/hpqtn/wh/collection1/&wt=javabin&version=2} {add=[2 (1438967178263003136)]} 0 4
[junit4:junit4] 2> 794217 T2812 C3908 P56990 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={wt=javabin&version=2} {add=[3 (1438967178268246016)]} 0 0
[junit4:junit4] 2> 794220 T2833 C3907 P38377 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:34506/hpqtn/wh/collection1/]
[junit4:junit4] 2> 794221 T2833 C3907 P38377 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:34506/hpqtn/wh/collection1/ against:[http://127.0.0.1:34506/hpqtn/wh/collection1/] result:true
[junit4:junit4] 2> 794221 T2833 C3907 P38377 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:52456/hpqtn/wh/collection1/ against:[http://127.0.0.1:34506/hpqtn/wh/collection1/] result:false
[junit4:junit4] 2> 794221 T2833 C3907 P38377 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:32859/hpqtn/wh/collection1/ against:[http://127.0.0.1:34506/hpqtn/wh/collection1/] result:false
[junit4:junit4] 2> 794224 T2883 C3906 P32859 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={distrib.from=http://127.0.0.1:38377/hpqtn/wh/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:34506/hpqtn/wh/collection1/&wt=javabin&version=2} {add=[3 (1438967178272440320)]} 0 1
[junit4:junit4] 2> 794224 T2865 C3909 P52456 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={distrib.from=http://127.0.0.1:38377/hpqtn/wh/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:34506/hpqtn/wh/collection1/&wt=javabin&version=2} {add=[3 (1438967178272440320)]} 0 1
[junit4:junit4] 2> 794225 T2833 C3907 P38377 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={distrib.from=http://127.0.0.1:32859/hpqtn/wh/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:34506/hpqtn/wh/collection1/&wt=javabin&version=2} {add=[3 (1438967178272440320)]} 0 5
[junit4:junit4] 2> 794225 T2881 C3906 P32859 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={test.distrib.skip.servers=http://127.0.0.1:34506/hpqtn/wh/collection1/&test.distrib.skip.servers=http://127.0.0.1:52456/hpqtn/wh/collection1/&wt=javabin&version=2} {add=[3]} 0 6
[junit4:junit4] 2> 794227 T2813 C3908 P56990 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 794242 T2813 C3908 P56990 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/control/data/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 794242 T2813 C3908 P56990 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 794395 T2813 C3908 P56990 oass.SolrIndexSearcher.<init> Opening Searcher@23627e main
[junit4:junit4] 2> 794396 T2813 C3908 P56990 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 794396 T2821 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@23627e main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 794396 T2813 C3908 P56990 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 169
[junit4:junit4] 2> 794398 T2846 C3910 P34506 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:38377/hpqtn/wh/collection1/, StdNode: http://127.0.0.1:34506/hpqtn/wh/collection1/, StdNode: http://127.0.0.1:52456/hpqtn/wh/collection1/, StdNode: http://127.0.0.1:32859/hpqtn/wh/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 794400 T2832 C3907 P38377 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 794400 T2880 C3906 P32859 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 794400 T2863 C3909 P52456 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 794400 T2847 C3910 P34506 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 794442 T2880 C3906 P32859 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 794443 T2880 C3906 P32859 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 794443 T2863 C3909 P52456 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty3/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 794443 T2832 C3907 P38377 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 794444 T2832 C3907 P38377 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 794443 T2847 C3910 P34506 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 794445 T2847 C3910 P34506 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 794444 T2880 C3906 P32859 oass.SolrIndexSearcher.<init> Opening Searcher@ef8a2f main
[junit4:junit4] 2> 794444 T2863 C3909 P52456 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 794446 T2880 C3906 P32859 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 794446 T2847 C3910 P34506 oass.SolrIndexSearcher.<init> Opening Searcher@ced34 main
[junit4:junit4] 2> 794446 T2832 C3907 P38377 oass.SolrIndexSearcher.<init> Opening Searcher@1a1dde1 main
[junit4:junit4] 2> 794447 T2863 C3909 P52456 oass.SolrIndexSearcher.<init> Opening Searcher@2f14a3 main
[junit4:junit4] 2> 794447 T2847 C3910 P34506 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 794446 T2891 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ef8a2f main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 794448 T2839 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1a1dde1 main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 794447 T2863 C3909 P52456 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 794447 T2855 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ced34 main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c2)}
[junit4:junit4] 2> 794447 T2832 C3907 P38377 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 794448 T2847 C3910 P34506 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 48
[junit4:junit4] 2> 794448 T2880 C3906 P32859 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 48
[junit4:junit4] 2> 794448 T2872 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2f14a3 main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 794449 T2832 C3907 P38377 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 49
[junit4:junit4] 2> 794450 T2863 C3909 P52456 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 50
[junit4:junit4] 2> 794450 T2846 C3910 P34506 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 52
[junit4:junit4] 2> 794450 T2795 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 794451 T2795 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 794452 T2795 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 794453 T2831 C3907 P38377 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 794455 T2848 C3910 P34506 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0
[junit4:junit4] 2> 794457 T2866 C3909 P52456 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 794459 T2883 C3906 P32859 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 794459 T2795 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 794463 T2864 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4] 2> 794463 T2864 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 794466 T2830 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4] 2> 794467 T2830 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38377/hpqtn/wh/collection1/
[junit4:junit4] 2> 794467 T2830 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:38377/hpqtn/wh START replicas=[http://127.0.0.1:34506/hpqtn/wh/collection1/, http://127.0.0.1:52456/hpqtn/wh/collection1/, http://127.0.0.1:32859/hpqtn/wh/collection1/] nUpdates=100
[junit4:junit4] 2> 794468 T2847 C3910 P34506 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 794469 T2881 C3906 P32859 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 794469 T2865 C3909 P52456 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 794469 T2830 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:38377/hpqtn/wh Received 3 versions from 127.0.0.1:34506/hpqtn/wh/collection1/
[junit4:junit4] 2> 794470 T2830 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:38377/hpqtn/wh Our versions are newer. ourLowThreshold=1438967178234691584 otherHigh=1438967178253565952
[junit4:junit4] 2> 794470 T2830 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:38377/hpqtn/wh Received 5 versions from 127.0.0.1:32859/hpqtn/wh/collection1/
[junit4:junit4] 2> 794470 T2830 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:38377/hpqtn/wh Our versions are newer. ourLowThreshold=1438967178234691584 otherHigh=1438967178263003136
[junit4:junit4] 2> 794471 T2830 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:38377/hpqtn/wh Received 5 versions from 127.0.0.1:52456/hpqtn/wh/collection1/
[junit4:junit4] 2> 794471 T2830 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:38377/hpqtn/wh Our versions are newer. ourLowThreshold=1438967178234691584 otherHigh=1438967178263003136
[junit4:junit4] 2> 794471 T2830 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:38377/hpqtn/wh DONE. sync succeeded
[junit4:junit4] 2> 794472 T2830 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 794472 T2830 oasc.SyncStrategy.syncToMe http://127.0.0.1:38377/hpqtn/wh/collection1/: try and ask http://127.0.0.1:34506/hpqtn/wh/collection1/ to sync
[junit4:junit4] 2> 794473 T2830 oasc.SyncStrategy.syncToMe http://127.0.0.1:38377/hpqtn/wh/collection1/: try and ask http://127.0.0.1:52456/hpqtn/wh/collection1/ to sync
[junit4:junit4] 2> 794473 T2830 oasc.SyncStrategy.syncToMe http://127.0.0.1:38377/hpqtn/wh/collection1/: try and ask http://127.0.0.1:32859/hpqtn/wh/collection1/ to sync
[junit4:junit4] 2> 794473 T2846 C3910 P34506 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:34506/hpqtn/wh START replicas=[http://127.0.0.1:38377/hpqtn/wh/collection1/] nUpdates=100
[junit4:junit4] 2> 794474 T2863 C3909 P52456 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52456/hpqtn/wh START replicas=[http://127.0.0.1:38377/hpqtn/wh/collection1/] nUpdates=100
[junit4:junit4] 2> 794474 T2882 C3906 P32859 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:32859/hpqtn/wh START replicas=[http://127.0.0.1:38377/hpqtn/wh/collection1/] nUpdates=100
[junit4:junit4] 2> 794475 T2833 C3907 P38377 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 794475 T2832 C3907 P38377 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 794475 T2831 C3907 P38377 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 794475 T2863 C3909 P52456 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:52456/hpqtn/wh Received 5 versions from 127.0.0.1:38377/hpqtn/wh/collection1/
[junit4:junit4] 2> 794476 T2846 C3910 P34506 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:34506/hpqtn/wh Received 5 versions from 127.0.0.1:38377/hpqtn/wh/collection1/
[junit4:junit4] 2> 794476 T2882 C3906 P32859 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:32859/hpqtn/wh Received 5 versions from 127.0.0.1:38377/hpqtn/wh/collection1/
[junit4:junit4] 2> 794476 T2846 C3910 P34506 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:34506/hpqtn/wh Requesting updates from 127.0.0.1:38377/hpqtn/wh/collection1/n=2 versions=[1438967178272440320, 1438967178263003136]
[junit4:junit4] 2> 794476 T2863 C3909 P52456 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:52456/hpqtn/wh Our versions are newer. ourLowThreshold=1438967178234691584 otherHigh=1438967178263003136
[junit4:junit4] 2> 794477 T2863 C3909 P52456 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52456/hpqtn/wh DONE. sync succeeded
[junit4:junit4] 2> 794477 T2882 C3906 P32859 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:32859/hpqtn/wh Our versions are newer. ourLowThreshold=1438967178234691584 otherHigh=1438967178263003136
[junit4:junit4] 2> 794477 T2863 C3909 P52456 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={sync=http://127.0.0.1:38377/hpqtn/wh/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=3
[junit4:junit4] 2> 794477 T2882 C3906 P32859 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:32859/hpqtn/wh DONE. sync succeeded
[junit4:junit4] 2> 794478 T2830 oasc.SyncStrategy.syncToMe http://127.0.0.1:38377/hpqtn/wh/collection1/: sync completed with http://127.0.0.1:52456/hpqtn/wh/collection1/
[junit4:junit4] 2> 794478 T2882 C3906 P32859 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={sync=http://127.0.0.1:38377/hpqtn/wh/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=4
[junit4:junit4] 2> 794478 T2833 C3907 P38377 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={distrib=false&getUpdates=1438967178272440320,1438967178263003136&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 794478 T2830 oasc.SyncStrategy.syncToMe http://127.0.0.1:38377/hpqtn/wh/collection1/: sync completed with http://127.0.0.1:32859/hpqtn/wh/collection1/
[junit4:junit4] 2> 794481 T2846 C3910 P34506 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1438967178263003136), 3 (1438967178272440320)]} 0 2
[junit4:junit4] 2> 794481 T2846 C3910 P34506 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:34506/hpqtn/wh DONE. sync succeeded
[junit4:junit4] 2> 794481 T2846 C3910 P34506 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={sync=http://127.0.0.1:38377/hpqtn/wh/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=8
[junit4:junit4] 2> 794482 T2830 oasc.SyncStrategy.syncToMe http://127.0.0.1:38377/hpqtn/wh/collection1/: sync completed with http://127.0.0.1:34506/hpqtn/wh/collection1/
[junit4:junit4] 2> 794482 T2830 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=16
[junit4:junit4] 2> 794482 T2864 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=19
[junit4:junit4] 2> 794483 T2795 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 794483 T2795 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 794484 T2795 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 794484 T2795 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 794486 T2810 C3908 P56990 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 794486 T2810 C3908 P56990 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 794487 T2810 C3908 P56990 oass.SolrIndexSearcher.<init> Opening Searcher@11dc707 main
[junit4:junit4] 2> 794487 T2810 C3908 P56990 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 794488 T2821 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@11dc707 main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 794489 T2810 C3908 P56990 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4] 2> 794491 T2848 C3910 P34506 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:38377/hpqtn/wh/collection1/, StdNode: http://127.0.0.1:34506/hpqtn/wh/collection1/, StdNode: http://127.0.0.1:52456/hpqtn/wh/collection1/, StdNode: http://127.0.0.1:32859/hpqtn/wh/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> ASYNC NEW_CORE C3911 name=collection1 org.apache.solr.core.SolrCore@937626 url=http://127.0.0.1:38377/hpqtn/wh/collection1 node=127.0.0.1:38377_hpqtn%2Fwh C3911_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:38377_hpqtn%2Fwh, base_url=http://127.0.0.1:38377/hpqtn/wh, leader=true}
[junit4:junit4] 2> 794499 T2831 C3911 P38377 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C3912 name=collection1 org.apache.solr.core.SolrCore@61dfb8 url=http://127.0.0.1:34506/hpqtn/wh/collection1 node=127.0.0.1:34506_hpqtn%2Fwh C3912_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:34506_hpqtn%2Fwh, base_url=http://127.0.0.1:34506/hpqtn/wh}
[junit4:junit4] 2> 794499 T2849 C3912 P34506 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C3913 name=collection1 org.apache.solr.core.SolrCore@10579f2 url=http://127.0.0.1:32859/hpqtn/wh/collection1 node=127.0.0.1:32859_hpqtn%2Fwh C3913_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:32859_hpqtn%2Fwh, base_url=http://127.0.0.1:32859/hpqtn/wh}
[junit4:junit4] 2> 794499 T2880 C3913 P32859 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 794500 T2880 C3913 P32859 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 794500 T2880 C3913 P32859 oass.SolrIndexSearcher.<init> Opening Searcher@1fcad0c main
[junit4:junit4] 2> 794501 T2880 C3913 P32859 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> ASYNC NEW_CORE C3914 name=collection1 org.apache.solr.core.SolrCore@71e6e6 url=http://127.0.0.1:52456/hpqtn/wh/collection1 node=127.0.0.1:52456_hpqtn%2Fwh C3914_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52456_hpqtn%2Fwh, base_url=http://127.0.0.1:52456/hpqtn/wh}
[junit4:junit4] 2> 794499 T2866 C3914 P52456 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 794501 T2866 C3914 P52456 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 794501 T2891 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1fcad0c main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 794499 T2831 C3911 P38377 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 794502 T2880 C3913 P32859 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4] 2> 794502 T2831 C3911 P38377 oass.SolrIndexSearcher.<init> Opening Searcher@be8790 main
[junit4:junit4] 2> 794502 T2866 C3914 P52456 oass.SolrIndexSearcher.<init> Opening Searcher@1c94041 main
[junit4:junit4] 2> 794503 T2831 C3911 P38377 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 794503 T2866 C3914 P52456 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 794503 T2839 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@be8790 main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 794504 T2872 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c94041 main{StandardDirectoryReader(segments_2:3:nrt _0(4.4):c4)}
[junit4:junit4] 2> 794504 T2831 C3911 P38377 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 794504 T2866 C3914 P52456 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 794542 T2849 C3912 P34506 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty2/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> 794543 T2849 C3912 P34506 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 794544 T2849 C3912 P34506 oass.SolrIndexSearcher.<init> Opening Searcher@1edd27e main
[junit4:junit4] 2> 794544 T2849 C3912 P34506 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 794545 T2855 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1edd27e main{StandardDirectoryReader(segments_3:5:nrt _0(4.4):c2 _1(4.4):c2)}
[junit4:junit4] 2> 794545 T2849 C3912 P34506 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 46
[junit4:junit4] 2> 794545 T2848 C3912 P34506 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 54
[junit4:junit4] 2> 794546 T2795 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 794548 T2833 C3911 P38377 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 794550 T2847 C3912 P34506 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 794552 T2865 C3914 P52456 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 794554 T2883 C3913 P32859 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 796555 T2795 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> check const of shard1
[junit4:junit4] 2> client0
[junit4:junit4] 2> PROPS:127.0.0.1:38377_hpqtn%2Fwh_collection1:{"shard":"shard1","state":"active","core":"collection1","collection":"collection1","node_name":"127.0.0.1:38377_hpqtn%2Fwh","base_url":"http://127.0.0.1:38377/hpqtn/wh","leader":"true"}
[junit4:junit4] 2> 796558 T2831 C3911 P38377 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client1
[junit4:junit4] 2> PROPS:127.0.0.1:34506_hpqtn%2Fwh_collection1:{"shard":"shard1","state":"active","core":"collection1","collection":"collection1","node_name":"127.0.0.1:34506_hpqtn%2Fwh","base_url":"http://127.0.0.1:34506/hpqtn/wh"}
[junit4:junit4] 2> 796559 T2848 C3912 P34506 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client2
[junit4:junit4] 2> PROPS:127.0.0.1:52456_hpqtn%2Fwh_collection1:{"shard":"shard1","state":"active","core":"collection1","collection":"collection1","node_name":"127.0.0.1:52456_hpqtn%2Fwh","base_url":"http://127.0.0.1:52456/hpqtn/wh"}
[junit4:junit4] 2> 796561 T2866 C3914 P52456 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client3
[junit4:junit4] 2> PROPS:127.0.0.1:32859_hpqtn%2Fwh_collection1:{"shard":"shard1","state":"active","core":"collection1","collection":"collection1","node_name":"127.0.0.1:32859_hpqtn%2Fwh","base_url":"http://127.0.0.1:32859/hpqtn/wh"}
[junit4:junit4] 2> 796563 T2880 C3913 P32859 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> 796565 T2881 C3913 P32859 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/select params={q=*:*&wt=javabin&version=2} hits=4 status=0 QTime=0
[junit4:junit4] 2> ASYNC NEW_CORE C3915 name=collection1 org.apache.solr.core.SolrCore@185dd28 url=http://127.0.0.1:56990/hpqtn/wh/collection1 node=127.0.0.1:56990_hpqtn%2Fwh C3915_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:56990_hpqtn%2Fwh, base_url=http://127.0.0.1:56990/hpqtn/wh, leader=true}
[junit4:junit4] 2> 796569 T2813 C3915 P56990 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={wt=javabin&version=2} {add=[4 (1438967180732399616)]} 0 2
[junit4:junit4] 2> 796572 T2830 C3911 P38377 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:34506/hpqtn/wh/collection1/]
[junit4:junit4] 2> 796572 T2830 C3911 P38377 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:34506/hpqtn/wh/collection1/ against:[http://127.0.0.1:34506/hpqtn/wh/collection1/] result:true
[junit4:junit4] 2> 796572 T2830 C3911 P38377 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:52456/hpqtn/wh/collection1/ against:[http://127.0.0.1:34506/hpqtn/wh/collection1/] result:false
[junit4:junit4] 2> 796573 T2830 C3911 P38377 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:32859/hpqtn/wh/collection1/ against:[http://127.0.0.1:34506/hpqtn/wh/collection1/] result:false
[junit4:junit4] 2> 796578 T2882 C3913 P32859 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={distrib.from=http://127.0.0.1:38377/hpqtn/wh/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:34506/hpqtn/wh/collection1/&wt=javabin&version=2} {add=[4 (1438967180738691072)]} 0 2
[junit4:junit4] 2> 796578 T2865 C3914 P52456 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={distrib.from=http://127.0.0.1:38377/hpqtn/wh/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:34506/hpqtn/wh/collection1/&wt=javabin&version=2} {add=[4 (1438967180738691072)]} 0 2
[junit4:junit4] 2> 796578 T2830 C3911 P38377 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={distrib.from=http://127.0.0.1:34506/hpqtn/wh/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:34506/hpqtn/wh/collection1/&wt=javabin&version=2} {add=[4 (1438967180738691072)]} 0 6
[junit4:junit4] 2> 796579 T2847 C3912 P34506 oasup.LogUpdateProcessor.finish [collection1] webapp=/hpqtn/wh path=/update params={test.distrib.skip.servers=http://127.0.0.1:34506/hpqtn/wh/collection1/&wt=javabin&version=2} {add=[4]} 0 9
[junit4:junit4] 2> 796579 T2795 oasc.ChaosMonkey.monkeyLog monkey: kill shard! 38377
[junit4:junit4] 2> 796600 T2795 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=9766679
[junit4:junit4] 2> 797457 T2818 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 797457 T2818 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:38377_hpqtn%2Fwh_collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:38377_hpqtn%2Fwh",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38377/hpqtn/wh"}
[junit4:junit4] 2> 797461 T2887 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 797461 T2837 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 797461 T2853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 797461 T2870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 797461 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 797461 T2817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 798602 T2795 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 798603 T2795 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 798604 T2795 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@937626
[junit4:junit4] 2> 798609 T2795 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=5,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 798609 T2795 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 798610 T2795 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 798610 T2795 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> C3911_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=down, core=collection1, collection=collection1, node_name=127.0.0.1:38377_hpqtn%2Fwh, base_url=http://127.0.0.1:38377/hpqtn/wh, leader=true}
[junit4:junit4] 2> 798622 T2795 C3911 P38377 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> 798623 T2795 C3911 P38377 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 798625 T2795 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 798625 T2795 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 798625 T2795 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1/index;done=false>>]
[junit4:junit4] 2> 798626 T2795 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1/index
[junit4:junit4] 2> 798626 T2795 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1;done=false>>]
[junit4:junit4] 2> 798626 T2795 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1372306017326/jetty1
[junit4:junit4] 2> 798633 T2817 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 798633 T2824 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 798634 T2795 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/hpqtn/wh,null}
[junit4:junit4] 2> 798634 T2870 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 798635 T2870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 798634 T2887 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 798635 T2887 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 798635 T2817 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 798635 T2824 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 798636 T2853 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 798637 T2853 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 798638 T2853 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 798638 T2853 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 798638 T2853 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:34506/hpqtn/wh/collection1/
[junit4:junit4] 2> 798638 T2853 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:34506/hpqtn/wh START replicas=[http://127.0.0.1:38377/hpqtn/wh/collection1/, http://127.0.0.1:52456/hpqtn/wh/collection1/, http://127.0.0.1:32859/hpqtn/wh/collection1/] nUpdates=100
[junit4:junit4] 2> 798639 T2853 oasu.PeerSync.handleResponse WARN PeerSync: core=collection1 url=http://127.0.0.1:34506/hpqtn/wh couldn't connect to http://127.0.0.1:38377/hpqtn/wh/collection1/, counting as success
[junit4:junit4] 2> 798640 T2883 C3913 P32859 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 798640 T2864 C3914 P52456 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 798641 T2853 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:34506/hpqtn/wh Received 6 versions from 127.0.0.1:32859/hpqtn/wh/collection1/
[junit4:junit4] 2> 798641 T2853 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:34506/hpqtn/wh Requesting updates from 127.0.0.1:32859/hpqtn/wh/collection1/n=1 versions=[1438967180738691072]
[junit4:junit4] 2> 798641 T2853 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:34506/hpqtn/wh Received 6 versions from 127.0.0.1:52456/hpqtn/wh/collection1/
[junit4:junit4] 2> 798641 T2853 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:34506/hpqtn/wh Our versions are newer. ourLowThreshold=1438967178234691584 otherHigh=1438967178272440320
[junit4:junit4] 2> 798643 T2880 C3913 P32859 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={distrib=false&getUpdates=1438967180738691072&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 798645 T2853 oasup.LogUpdateProcessor.finish [collection1] {add=[4 (1438967180738691072)]} 0 2
[junit4:junit4] 2> 798645 T2853 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:34506/hpqtn/wh DONE. sync succeeded
[junit4:junit4] 2> 798645 T2853 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 798645 T2853 oasc.SyncStrategy.syncToMe http://127.0.0.1:34506/hpqtn/wh/collection1/: try and ask http://127.0.0.1:38377/hpqtn/wh/collection1/ to sync
[junit4:junit4] 2> 798646 T2853 oasc.SyncStrategy.syncToMe http://127.0.0.1:34506/hpqtn/wh/collection1/: try and ask http://127.0.0.1:52456/hpqtn/wh/collection1/ to sync
[junit4:junit4] 2> 798646 T2853 oasc.SyncStrategy.syncToMe http://127.0.0.1:34506/hpqtn/wh/collection1/: try and ask http://127.0.0.1:32859/hpqtn/wh/collection1/ to sync
[junit4:junit4] 2> 798646 T2853 oasc.SolrException.log ERROR Sync request error: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:38377/hpqtn/wh/collection1
[junit4:junit4] 2> 798646 T2853 oasc.SyncStrategy.syncToMe http://127.0.0.1:34506/hpqtn/wh/collection1/: Sync failed - asking replica (http://127.0.0.1:38377/hpqtn/wh/collection1/) to recover.
[junit4:junit4] 2> 798647 T2920 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 798647 T2866 C3914 P52456 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52456/hpqtn/wh START replicas=[http://127.0.0.1:34506/hpqtn/wh/collection1/] nUpdates=100
[junit4:junit4] 2> 798648 T2882 C3913 P32859 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:32859/hpqtn/wh START replicas=[http://127.0.0.1:34506/hpqtn/wh/collection1/] nUpdates=100
[junit4:junit4] 2> 798649 T2847 C3912 P34506 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 798649 T2846 C3912 P34506 oasc.SolrCore.execute [collection1] webapp=/hpqtn/wh path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 798649 T2882 C3913 P32859 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:32859/hpqtn/wh Received 6 versions from 127.0.0.1:34506/hpqtn/wh/collection1/
[junit4:junit4] 2> 798649 T2866 C3914 P52456 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:52456/hpqtn/wh Received 6 versions from 127.0.0.1:34506/hpqtn/wh/collection1/
[junit4:junit4] 2> 798650 T2882 C3913 P32859 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:32859/hpqtn/wh Our versions are newer. ourLowThreshold=1438967178244128768 otherHigh=1438967178272440320
[junit4:junit4] 2> 798650 T2920 oasc.SolrException.log ERROR http://127.0.0.1:34506/hpqtn/wh/collection1/: Could not tell a replica to recover:org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:38377/hpqtn/wh
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
[junit4:junit4]
[...truncated too long message...]
0.1:56990_hpqtn%2Fwh",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:56990/hpqtn/wh",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:32859_hpqtn%2Fwh (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:52456_hpqtn%2Fwh (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:38377_hpqtn%2Fwh (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:56990_hpqtn%2Fwh (0)
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/collections (2)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89935447151673354-127.0.0.1:52456_hpqtn%2Fwh_collection1-n_0000000002 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89935447151673356-127.0.0.1:32859_hpqtn%2Fwh_collection1-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89935447151673358-127.0.0.1:38377_hpqtn%2Fwh_collection1-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (1)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:52456_hpqtn%2Fwh",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:52456/hpqtn/wh"}
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89935447151673347-127.0.0.1:56990_hpqtn%2Fwh_collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:56990_hpqtn%2Fwh",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:56990/hpqtn/wh"}
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/election (4)
[junit4:junit4] 1> /solr/overseer_elect/election/89935447151673347-127.0.0.1:56990_hpqtn%2Fwh-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89935447151673358-127.0.0.1:38377_hpqtn%2Fwh-n_0000000005 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89935447151673354-127.0.0.1:52456_hpqtn%2Fwh-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89935447151673356-127.0.0.1:32859_hpqtn%2Fwh-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89935447151673347-127.0.0.1:56990_hpqtn%2Fwh-n_0000000000"}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=4B7778AE2F424EAE -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_HN -Dtests.timezone=America/Cayenne -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE 98.7s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:38377/hpqtn/wh/collection1lastClient and got 250 from http://127.0.0.1:52456/hpqtn/wh/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([4B7778AE2F424EAE:CA91F6B6581D2E92]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 870211 T2795 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 98742 T2794 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 870270 T2870 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 870271 T2870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 870271 T2870 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 871343 T2887 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 871344 T2887 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 871344 T2887 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=88), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=88)), sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=es_HN, timezone=America/Cayenne
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation 1.7.0_25 (32-bit)/cpus=8,threads=1,free=73707616,total=278847488
[junit4:junit4] 2> NOTE: All tests run in this JVM: [FieldMutatingUpdateProcessorTest, SolrIndexSplitterTest, LukeRequestHandlerTest, TestFuzzyAnalyzedSuggestions, ResourceLoaderTest, TestSolrQueryParserResource, TestFaceting, TestFieldTypeCollectionResource, TestSolrCoreProperties, TestDocSet, TestDynamicFieldCollectionResource, SoftAutoCommitTest, TestSolrIndexConfig, BasicFunctionalityTest, DocValuesTest, UnloadDistributedZkTest, TermsComponentTest, TestMaxScoreQueryParser, TestFastLRUCache, TestSweetSpotSimilarityFactory, CoreAdminHandlerTest, OverseerCollectionProcessorTest, RAMDirectoryFactoryTest, TestShardHandlerFactory, FileBasedSpellCheckerTest, TestRemoteStreaming, AnalysisAfterCoreReloadTest, TestReplicationHandler, TestFastWriter, PreAnalyzedFieldTest, QueryEqualityTest, TermVectorComponentTest, TestSchemaNameResource, TestRandomDVFaceting, TestWriterPerf, TestJoin, ParsingFieldUpdateProcessorsTest, TestFieldResource, TestTrie, TestCharFilters, AlternateDirectoryTest, DistributedTermsComponentTest, SystemInfoHandlerTest, TestGroupingSearch, TestBadConfig, MinimalSchemaTest, SolrTestCaseJ4Test, JsonLoaderTest, DirectUpdateHandlerOptimizeTest, TestValueSourceCache, TestSolrDeletionPolicy2, StatelessScriptUpdateProcessorFactoryTest, PeerSyncTest, MBeansHandlerTest, TestQueryUtils, TestLRUCache, SolrInfoMBeanTest, TestFieldTypeResource, SynonymTokenizerTest, TestExtendedDismaxParser, SchemaVersionSpecificBehaviorTest, IndexSchemaTest, TestIBSimilarityFactory, ShowFileRequestHandlerTest, SuggesterTest, TestSolrQueryParser, LoggingHandlerTest, TestCopyFieldCollectionResource, SOLR749Test, TestManagedSchemaFieldResource, TestAddFieldRealTimeGet, TestStressRecovery, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, SyncSliceTest]
[junit4:junit4] Completed on J0 in 100.01s, 1 test, 1 failure <<< FAILURES!
[...truncated 455 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1246: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:890: There were test failures: 299 suites, 1289 tests, 1 failure, 13 ignored (7 assumptions)
Total time: 42 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_25 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure