You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/22 09:09:45 UTC
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b93) - Build #
6251 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6251/
Java: 64bit/jdk1.8.0-ea-b93 -XX:-UseCompressedOops -XX:+UseParallelGC
1 tests failed.
FAILED: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:51741/collection1lastClient and got 246 from http://127.0.0.1:45589/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:51741/collection1lastClient and got 246 from http://127.0.0.1:45589/collection1
at __randomizedtesting.SeedInfo.seed([39BC0AE900E83EF:827D4EB6E751E3D3]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:491)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9702 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4] 2> 316187 T1460 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 316192 T1460 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1371884297708
[junit4:junit4] 2> 316192 T1460 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 316193 T1461 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 316293 T1460 oasc.ZkTestServer.run start zk server on port:55022
[junit4:junit4] 2> 316294 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 316312 T1467 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d2da166 name:ZooKeeperConnection Watcher:127.0.0.1:55022 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 316312 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 316313 T1460 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 316317 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 316318 T1469 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33ea58b3 name:ZooKeeperConnection Watcher:127.0.0.1:55022/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 316318 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 316319 T1460 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 316321 T1460 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 316323 T1460 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 316325 T1460 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 316327 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 316327 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 316331 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 316331 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 316333 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 316334 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 316336 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 316336 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 316338 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 316338 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 316340 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 316340 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 316342 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 316343 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 316345 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 316345 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 316347 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 316347 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 316418 T1460 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 316438 T1460 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55891
[junit4:junit4] 2> 316439 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 316439 T1460 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 316440 T1460 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371884297866
[junit4:junit4] 2> 316440 T1460 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371884297866/solr.xml
[junit4:junit4] 2> 316440 T1460 oasc.CoreContainer.<init> New CoreContainer 1106761978
[junit4:junit4] 2> 316441 T1460 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371884297866/'
[junit4:junit4] 2> 316441 T1460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371884297866/'
[junit4:junit4] 2> 316483 T1460 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 316483 T1460 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 316483 T1460 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 316483 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 316484 T1460 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 316484 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 316484 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 316484 T1460 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 316485 T1460 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 316485 T1460 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 316488 T1460 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 316488 T1460 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55022/solr
[junit4:junit4] 2> 316489 T1460 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 316489 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 316491 T1480 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@242056d7 name:ZooKeeperConnection Watcher:127.0.0.1:55022 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 316491 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 316493 T1460 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 316496 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 316498 T1482 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12996d13 name:ZooKeeperConnection Watcher:127.0.0.1:55022/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 316498 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 316499 T1460 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 316502 T1460 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 316504 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 316506 T1460 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55891_
[junit4:junit4] 2> 316506 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55891_
[junit4:junit4] 2> 316509 T1460 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 316513 T1460 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 316515 T1460 oasc.Overseer.start Overseer (id=89907809334722563-127.0.0.1:55891_-n_0000000000) starting
[junit4:junit4] 2> 316519 T1460 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 316525 T1460 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 316526 T1484 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 316528 T1460 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 316530 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 316536 T1483 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 316536 T1485 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 316536 T1485 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 316542 T1483 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 316542 T1483 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55891",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:55891_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 316542 T1483 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 316543 T1483 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 316546 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 317537 T1485 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371884297866/collection1
[junit4:junit4] 2> 317538 T1485 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 317538 T1485 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 317539 T1485 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 317540 T1485 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371884297866/collection1/'
[junit4:junit4] 2> 317541 T1485 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1371884297866/collection1/lib/README' to classloader
[junit4:junit4] 2> 317541 T1485 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1371884297866/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 317571 T1485 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 317607 T1485 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 317609 T1485 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 317613 T1485 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 317947 T1485 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 317953 T1485 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 317956 T1485 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 317967 T1485 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 317969 T1485 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 317972 T1485 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 317973 T1485 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 317973 T1485 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 317973 T1485 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 317974 T1485 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 317974 T1485 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 317974 T1485 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 317974 T1485 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371884297866/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371884297708/control/data/
[junit4:junit4] 2> 317974 T1485 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4e604ca9
[junit4:junit4] 2> 317975 T1485 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 317975 T1485 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/control/data
[junit4:junit4] 2> 317975 T1485 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371884297708/control/data/index/
[junit4:junit4] 2> 317976 T1485 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371884297708/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 317976 T1485 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/control/data/index
[junit4:junit4] 2> 317979 T1485 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 317979 T1485 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 317981 T1485 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 317981 T1485 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 317982 T1485 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 317982 T1485 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 317982 T1485 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 317982 T1485 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 317983 T1485 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 317983 T1485 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 317983 T1485 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 317985 T1485 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 317988 T1485 oass.SolrIndexSearcher.<init> Opening Searcher@10c8fdd5 main
[junit4:junit4] 2> 317988 T1485 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 317988 T1485 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 317991 T1486 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@10c8fdd5 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 317992 T1485 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 317993 T1485 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55891 collection:control_collection shard:shard1
[junit4:junit4] 2> 317993 T1485 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 317999 T1485 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 318001 T1485 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 318001 T1485 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 318001 T1485 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55891/collection1/
[junit4:junit4] 2> 318002 T1485 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 318002 T1485 oasc.SyncStrategy.syncToMe http://127.0.0.1:55891/collection1/ has no replicas
[junit4:junit4] 2> 318002 T1485 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55891/collection1/
[junit4:junit4] 2> 318002 T1485 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 318050 T1483 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 318055 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 318106 T1485 oasc.ZkController.register We are http://127.0.0.1:55891/collection1/ and leader is http://127.0.0.1:55891/collection1/
[junit4:junit4] 2> 318107 T1485 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55891
[junit4:junit4] 2> 318107 T1485 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 318107 T1485 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 318107 T1485 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 318109 T1485 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 318110 T1460 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 318110 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 318110 T1460 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 318113 T1460 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 318114 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 318116 T1489 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@547a5bb9 name:ZooKeeperConnection Watcher:127.0.0.1:55022/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 318116 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 318117 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 318119 T1460 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 318182 T1460 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 318184 T1460 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51741
[junit4:junit4] 2> 318184 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 318185 T1460 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 318185 T1460 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371884299635
[junit4:junit4] 2> 318185 T1460 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371884299635/solr.xml
[junit4:junit4] 2> 318185 T1460 oasc.CoreContainer.<init> New CoreContainer 156946197
[junit4:junit4] 2> 318186 T1460 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371884299635/'
[junit4:junit4] 2> 318186 T1460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371884299635/'
[junit4:junit4] 2> 318223 T1460 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 318223 T1460 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 318223 T1460 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 318223 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 318224 T1460 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 318224 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 318224 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 318224 T1460 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 318224 T1460 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 318225 T1460 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 318227 T1460 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 318227 T1460 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55022/solr
[junit4:junit4] 2> 318228 T1460 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 318228 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 318230 T1500 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61e38e66 name:ZooKeeperConnection Watcher:127.0.0.1:55022 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 318230 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 318232 T1460 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 318234 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 318236 T1502 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2171f90a name:ZooKeeperConnection Watcher:127.0.0.1:55022/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 318236 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 318239 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 319241 T1460 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51741_
[junit4:junit4] 2> 319242 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51741_
[junit4:junit4] 2> 319245 T1482 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 319246 T1502 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 319246 T1489 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 319246 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 319253 T1503 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 319254 T1503 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 319559 T1483 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 319559 T1483 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55891",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:55891_",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 319563 T1483 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51741",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:51741_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 319563 T1483 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 319563 T1483 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 319566 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 319566 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 319566 T1489 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 320256 T1503 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371884299635/collection1
[junit4:junit4] 2> 320256 T1503 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 320256 T1503 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 320257 T1503 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 320258 T1503 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371884299635/collection1/'
[junit4:junit4] 2> 320258 T1503 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1371884299635/collection1/lib/README' to classloader
[junit4:junit4] 2> 320258 T1503 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1371884299635/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 320289 T1503 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 320318 T1503 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 320320 T1503 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 320325 T1503 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 320637 T1503 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 320642 T1503 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 320646 T1503 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 320669 T1503 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 320672 T1503 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 320675 T1503 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 320677 T1503 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 320677 T1503 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 320677 T1503 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 320678 T1503 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 320678 T1503 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 320679 T1503 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 320679 T1503 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371884299635/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/
[junit4:junit4] 2> 320679 T1503 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4e604ca9
[junit4:junit4] 2> 320679 T1503 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 320680 T1503 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1
[junit4:junit4] 2> 320680 T1503 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index/
[junit4:junit4] 2> 320680 T1503 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 320681 T1503 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index
[junit4:junit4] 2> 320685 T1503 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 320686 T1503 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 320688 T1503 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 320688 T1503 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 320689 T1503 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 320690 T1503 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 320690 T1503 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 320690 T1503 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 320691 T1503 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 320691 T1503 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 320692 T1503 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 320694 T1503 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 320698 T1503 oass.SolrIndexSearcher.<init> Opening Searcher@35f237e2 main
[junit4:junit4] 2> 320698 T1503 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 320699 T1503 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 320703 T1504 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@35f237e2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 320705 T1503 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 320705 T1503 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51741 collection:collection1 shard:shard1
[junit4:junit4] 2> 320706 T1503 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 320711 T1503 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 320713 T1503 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 320713 T1503 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 320713 T1503 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51741/collection1/
[junit4:junit4] 2> 320713 T1503 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 320714 T1503 oasc.SyncStrategy.syncToMe http://127.0.0.1:51741/collection1/ has no replicas
[junit4:junit4] 2> 320714 T1503 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51741/collection1/
[junit4:junit4] 2> 320714 T1503 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 321095 T1483 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 321103 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 321103 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 321104 T1489 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 321119 T1503 oasc.ZkController.register We are http://127.0.0.1:51741/collection1/ and leader is http://127.0.0.1:51741/collection1/
[junit4:junit4] 2> 321119 T1503 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51741
[junit4:junit4] 2> 321120 T1503 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 321120 T1503 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 321120 T1503 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 321121 T1503 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 321122 T1460 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 321122 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 321123 T1460 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 321198 T1460 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 321200 T1460 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59479
[junit4:junit4] 2> 321200 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 321200 T1460 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 321201 T1460 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371884302641
[junit4:junit4] 2> 321201 T1460 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371884302641/solr.xml
[junit4:junit4] 2> 321201 T1460 oasc.CoreContainer.<init> New CoreContainer 1036469393
[junit4:junit4] 2> 321201 T1460 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371884302641/'
[junit4:junit4] 2> 321202 T1460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371884302641/'
[junit4:junit4] 2> 321245 T1460 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 321246 T1460 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 321246 T1460 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 321246 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 321246 T1460 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 321247 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 321247 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 321247 T1460 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 321247 T1460 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 321247 T1460 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 321250 T1460 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 321251 T1460 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55022/solr
[junit4:junit4] 2> 321251 T1460 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 321252 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 321255 T1516 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f63824c name:ZooKeeperConnection Watcher:127.0.0.1:55022 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 321255 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 321258 T1460 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 321262 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 321263 T1518 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@792ae023 name:ZooKeeperConnection Watcher:127.0.0.1:55022/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 321264 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 321268 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 322273 T1460 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59479_
[junit4:junit4] 2> 322275 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59479_
[junit4:junit4] 2> 322302 T1502 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 322303 T1482 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 322302 T1518 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 322303 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 322303 T1489 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 322303 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 322303 T1489 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 322308 T1519 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 322308 T1519 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 322609 T1483 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 322610 T1483 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51741",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:51741_",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 322613 T1483 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59479",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:59479_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 322613 T1483 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 322613 T1483 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 322615 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 322616 T1489 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 322615 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 322615 T1518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 323310 T1519 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371884302641/collection1
[junit4:junit4] 2> 323310 T1519 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 323311 T1519 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 323311 T1519 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 323312 T1519 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371884302641/collection1/'
[junit4:junit4] 2> 323312 T1519 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1371884302641/collection1/lib/README' to classloader
[junit4:junit4] 2> 323313 T1519 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1371884302641/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 323335 T1519 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 323362 T1519 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 323364 T1519 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 323367 T1519 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 323689 T1519 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 323695 T1519 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 323698 T1519 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 323709 T1519 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 323712 T1519 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 323714 T1519 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 323715 T1519 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 323716 T1519 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 323716 T1519 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 323717 T1519 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 323717 T1519 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 323717 T1519 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 323717 T1519 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371884302641/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/
[junit4:junit4] 2> 323717 T1519 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4e604ca9
[junit4:junit4] 2> 323717 T1519 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 323718 T1519 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2
[junit4:junit4] 2> 323718 T1519 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index/
[junit4:junit4] 2> 323718 T1519 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 323718 T1519 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index
[junit4:junit4] 2> 323721 T1519 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 323721 T1519 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 323723 T1519 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 323723 T1519 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 323723 T1519 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 323724 T1519 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 323724 T1519 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 323724 T1519 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 323724 T1519 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 323725 T1519 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 323725 T1519 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 323726 T1519 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 323729 T1519 oass.SolrIndexSearcher.<init> Opening Searcher@3439953f main
[junit4:junit4] 2> 323730 T1519 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 323730 T1519 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 323734 T1520 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3439953f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 323736 T1519 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 323736 T1519 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59479 collection:collection1 shard:shard1
[junit4:junit4] 2> 323739 T1519 oasc.ZkController.register We are http://127.0.0.1:59479/collection1/ and leader is http://127.0.0.1:51741/collection1/
[junit4:junit4] 2> 323739 T1519 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59479
[junit4:junit4] 2> 323739 T1519 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 323740 T1519 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C255 name=collection1 org.apache.solr.core.SolrCore@71243dc0 url=http://127.0.0.1:59479/collection1 node=127.0.0.1:59479_ C255_STATE=coll:collection1 core:collection1 props:{collection=collection1, base_url=http://127.0.0.1:59479, node_name=127.0.0.1:59479_, state=down, core=collection1}
[junit4:junit4] 2> 323740 T1521 C255 P59479 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 323741 T1521 C255 P59479 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 323741 T1519 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 323741 T1521 C255 P59479 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 323741 T1521 C255 P59479 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 323742 T1460 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 323742 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 323742 T1521 C255 P59479 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 323742 T1460 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 323747 T1496 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 323808 T1460 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 323810 T1460 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45589
[junit4:junit4] 2> 323811 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 323811 T1460 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 323811 T1460 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371884305262
[junit4:junit4] 2> 323812 T1460 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371884305262/solr.xml
[junit4:junit4] 2> 323812 T1460 oasc.CoreContainer.<init> New CoreContainer 1910853850
[junit4:junit4] 2> 323812 T1460 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371884305262/'
[junit4:junit4] 2> 323812 T1460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371884305262/'
[junit4:junit4] 2> 323849 T1460 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 323850 T1460 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 323850 T1460 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 323850 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 323850 T1460 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 323850 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 323851 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 323851 T1460 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 323851 T1460 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 323851 T1460 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 323854 T1460 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 323854 T1460 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55022/solr
[junit4:junit4] 2> 323855 T1460 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 323855 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 323857 T1533 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6155352c name:ZooKeeperConnection Watcher:127.0.0.1:55022 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 323858 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 323859 T1460 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 323862 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 323863 T1535 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2811a272 name:ZooKeeperConnection Watcher:127.0.0.1:55022/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 323864 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 323866 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 324120 T1483 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 324121 T1483 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59479",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:59479_",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 324125 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 324126 T1518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 324126 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 324126 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 324126 T1489 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 324748 T1496 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 324748 T1496 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={action=PREPRECOVERY&version=2&checkLive=true&wt=javabin&nodeName=127.0.0.1:59479_&state=recovering&core=collection1&onlyIfLeader=true&coreNodeName=3} status=0 QTime=1001
[junit4:junit4] 2> 324868 T1460 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45589_
[junit4:junit4] 2> 324871 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45589_
[junit4:junit4] 2> 324872 T1518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 324872 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 324873 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 324872 T1489 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 324873 T1502 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 324873 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 324875 T1482 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 324875 T1489 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 324875 T1518 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 324876 T1535 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 324881 T1536 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 324881 T1536 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 325630 T1483 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 325631 T1483 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45589",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:45589_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 325631 T1483 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 325631 T1483 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 325641 T1489 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 325641 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 325641 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 325641 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 325641 T1518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 325883 T1536 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371884305262/collection1
[junit4:junit4] 2> 325883 T1536 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 325884 T1536 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 325884 T1536 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 325885 T1536 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371884305262/collection1/'
[junit4:junit4] 2> 325885 T1536 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1371884305262/collection1/lib/README' to classloader
[junit4:junit4] 2> 325886 T1536 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1371884305262/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 325912 T1536 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 325948 T1536 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 325949 T1536 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 325952 T1536 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 326239 T1536 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 326246 T1536 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 326250 T1536 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 326270 T1536 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 326273 T1536 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 326277 T1536 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 326279 T1536 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 326279 T1536 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 326280 T1536 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 326281 T1536 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 326281 T1536 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 326281 T1536 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 326281 T1536 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371884305262/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/
[junit4:junit4] 2> 326281 T1536 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4e604ca9
[junit4:junit4] 2> 326282 T1536 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 326282 T1536 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3
[junit4:junit4] 2> 326283 T1536 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index/
[junit4:junit4] 2> 326283 T1536 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 326283 T1536 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index
[junit4:junit4] 2> 326293 T1536 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 326293 T1536 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 326297 T1536 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 326297 T1536 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 326298 T1536 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 326299 T1536 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 326299 T1536 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 326299 T1536 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 326300 T1536 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 326300 T1536 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 326301 T1536 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 326303 T1536 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 326307 T1536 oass.SolrIndexSearcher.<init> Opening Searcher@d0f0c94 main
[junit4:junit4] 2> 326308 T1536 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 326308 T1536 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 326315 T1537 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d0f0c94 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 326317 T1536 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 326317 T1536 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45589 collection:collection1 shard:shard1
[junit4:junit4] 2> 326320 T1536 oasc.ZkController.register We are http://127.0.0.1:45589/collection1/ and leader is http://127.0.0.1:51741/collection1/
[junit4:junit4] 2> 326321 T1536 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45589
[junit4:junit4] 2> 326321 T1536 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 326321 T1536 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C256 name=collection1 org.apache.solr.core.SolrCore@2a6509a2 url=http://127.0.0.1:45589/collection1 node=127.0.0.1:45589_ C256_STATE=coll:collection1 core:collection1 props:{collection=collection1, base_url=http://127.0.0.1:45589, node_name=127.0.0.1:45589_, state=down, core=collection1}
[junit4:junit4] 2> 326321 T1538 C256 P45589 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 326322 T1536 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 326322 T1538 C256 P45589 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 326322 T1538 C256 P45589 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 326322 T1538 C256 P45589 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 326325 T1460 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 326326 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 326326 T1538 C256 P45589 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 326326 T1460 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 326335 T1496 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 326425 T1460 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 326427 T1460 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59727
[junit4:junit4] 2> 326428 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 326428 T1460 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 326428 T1460 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371884307847
[junit4:junit4] 2> 326428 T1460 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371884307847/solr.xml
[junit4:junit4] 2> 326429 T1460 oasc.CoreContainer.<init> New CoreContainer 1811398186
[junit4:junit4] 2> 326429 T1460 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371884307847/'
[junit4:junit4] 2> 326429 T1460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371884307847/'
[junit4:junit4] 2> 326476 T1460 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 326476 T1460 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 326476 T1460 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 326477 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 326477 T1460 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 326477 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 326477 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 326477 T1460 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 326478 T1460 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 326478 T1460 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 326480 T1460 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 326481 T1460 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55022/solr
[junit4:junit4] 2> 326481 T1460 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 326482 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 326484 T1550 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72992c16 name:ZooKeeperConnection Watcher:127.0.0.1:55022 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 326484 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 326486 T1460 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 326488 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 326490 T1552 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6044b6f1 name:ZooKeeperConnection Watcher:127.0.0.1:55022/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 326490 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 326495 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C257 name=collection1 org.apache.solr.core.SolrCore@71243dc0 url=http://127.0.0.1:59479/collection1 node=127.0.0.1:59479_ C257_STATE=coll:collection1 core:collection1 props:{collection=collection1, base_url=http://127.0.0.1:59479, node_name=127.0.0.1:59479_, state=recovering, shard=shard1, core=collection1}
[junit4:junit4] 2> 326749 T1521 C257 P59479 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:51741/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 326749 T1521 C257 P59479 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59479 START replicas=[http://127.0.0.1:51741/collection1/] nUpdates=100
[junit4:junit4] 2> 326750 T1521 C257 P59479 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 326750 T1521 C257 P59479 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 326750 T1521 C257 P59479 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 326750 T1521 C257 P59479 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 326750 T1521 C257 P59479 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 326751 T1521 C257 P59479 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:51741/collection1/. core=collection1
[junit4:junit4] 2> 326751 T1521 C257 P59479 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C258 name=collection1 org.apache.solr.core.SolrCore@51713d58 url=http://127.0.0.1:51741/collection1 node=127.0.0.1:51741_ C258_STATE=coll:collection1 core:collection1 props:{collection=collection1, base_url=http://127.0.0.1:51741, node_name=127.0.0.1:51741_, state=active, shard=shard1, core=collection1, leader=true}
[junit4:junit4] 2> 326756 T1495 C258 P51741 oasc.SolrCore.execute [collection1] webapp= path=/get params={version=2&distrib=false&getVersions=100&wt=javabin&qt=/get} status=0 QTime=0
[junit4:junit4] 2> 326758 T1498 C258 P51741 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 326760 T1498 C258 P51741 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 326761 T1498 C258 P51741 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 326781 T1498 C258 P51741 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 326782 T1498 C258 P51741 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 326782 T1498 C258 P51741 oass.SolrIndexSearcher.<init> Opening Searcher@42a328d0 realtime
[junit4:junit4] 2> 326782 T1498 C258 P51741 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 326783 T1498 C258 P51741 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&commit_end_point=true&waitSearcher=true&wt=javabin&openSearcher=false&commit=true&softCommit=false} {commit=} 0 25
[junit4:junit4] 2> 326784 T1521 C257 P59479 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 326785 T1521 C257 P59479 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 326788 T1497 C258 P51741 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 326788 T1497 C258 P51741 oasc.SolrCore.execute [collection1] webapp= path=/replication params={version=2&command=indexversion&qt=/replication&wt=javabin} status=0 QTime=2
[junit4:junit4] 2> 326789 T1521 C257 P59479 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 326789 T1521 C257 P59479 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 326789 T1521 C257 P59479 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 326791 T1498 C258 P51741 oasc.SolrCore.execute [collection1] webapp= path=/replication params={version=2&command=filelist&qt=/replication&wt=javabin&generation=2} status=0 QTime=0
[junit4:junit4] 2> 326792 T1521 C257 P59479 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 326792 T1521 C257 P59479 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index.20130622025828308
[junit4:junit4] 2> 326793 T1521 C257 P59479 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index.20130622025828308 lockFactory=org.apache.lucene.store.NativeFSLockFactory@60dcd0d6 fullCopy=false
[junit4:junit4] 2> 326797 T1495 C258 P51741 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 326799 T1521 C257 P59479 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 326802 T1521 C257 P59479 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 326802 T1521 C257 P59479 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 326805 T1521 C257 P59479 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 326805 T1521 C257 P59479 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 326806 T1521 C257 P59479 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 326806 T1521 C257 P59479 oass.SolrIndexSearcher.<init> Opening Searcher@a1ec381 main
[junit4:junit4] 2> 326807 T1520 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a1ec381 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 326807 T1521 C257 P59479 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index.20130622025828308 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index.20130622025828308;done=true>>]
[junit4:junit4] 2> 326807 T1521 C257 P59479 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index.20130622025828308
[junit4:junit4] 2> 326808 T1521 C257 P59479 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index.20130622025828308
[junit4:junit4] 2> 326808 T1521 C257 P59479 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 326808 T1521 C257 P59479 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 326808 T1521 C257 P59479 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 326808 T1521 C257 P59479 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 326810 T1521 C257 P59479 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 327146 T1483 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 327147 T1483 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45589",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:45589_",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 327150 T1483 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59479",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:59479_",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 327153 T1489 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 327154 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 327154 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 327154 T1552 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 327154 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 327154 T1518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 327336 T1496 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 327336 T1496 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={action=PREPRECOVERY&version=2&checkLive=true&wt=javabin&nodeName=127.0.0.1:45589_&state=recovering&core=collection1&onlyIfLeader=true&coreNodeName=4} status=0 QTime=1001
[junit4:junit4] 2> 327498 T1460 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59727_
[junit4:junit4] 2> 327511 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59727_
[junit4:junit4] 2> 327514 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 327514 T1552 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 327514 T1489 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 327515 T1552 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 327515 T1482 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 327515 T1518 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 327515 T1502 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 327515 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 327515 T1489 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 327515 T1518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 327515 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 327516 T1535 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 327520 T1555 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 327520 T1555 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 328660 T1483 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 328661 T1483 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59727",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:59727_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 328661 T1483 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 328661 T1483 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 328665 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 328665 T1489 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 328665 T1518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 328665 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 328665 T1552 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 328665 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> ASYNC NEW_CORE C259 name=collection1 org.apache.solr.core.SolrCore@2a6509a2 url=http://127.0.0.1:45589/collection1 node=127.0.0.1:45589_ C259_STATE=coll:collection1 core:collection1 props:{collection=collection1, base_url=http://127.0.0.1:45589, node_name=127.0.0.1:45589_, state=recovering, shard=shard1, core=collection1}
[junit4:junit4] 2> 329337 T1538 C259 P45589 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:51741/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 329337 T1538 C259 P45589 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45589 START replicas=[http://127.0.0.1:51741/collection1/] nUpdates=100
[junit4:junit4] 2> 329338 T1538 C259 P45589 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 329338 T1538 C259 P45589 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 329338 T1538 C259 P45589 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 329338 T1538 C259 P45589 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 329338 T1538 C259 P45589 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 329339 T1538 C259 P45589 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:51741/collection1/. core=collection1
[junit4:junit4] 2> 329339 T1538 C259 P45589 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 329340 T1496 C258 P51741 oasc.SolrCore.execute [collection1] webapp= path=/get params={version=2&distrib=false&getVersions=100&wt=javabin&qt=/get} status=0 QTime=0
[junit4:junit4] 2> 329342 T1497 C258 P51741 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 329351 T1497 C258 P51741 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> 329352 T1497 C258 P51741 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 329352 T1497 C258 P51741 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 329352 T1497 C258 P51741 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&commit_end_point=true&waitSearcher=true&wt=javabin&openSearcher=false&commit=true&softCommit=false} {commit=} 0 10
[junit4:junit4] 2> 329353 T1538 C259 P45589 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 329353 T1538 C259 P45589 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 329355 T1496 C258 P51741 oasc.SolrCore.execute [collection1] webapp= path=/replication params={version=2&command=indexversion&qt=/replication&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 329355 T1538 C259 P45589 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4] 2> 329355 T1538 C259 P45589 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 329355 T1538 C259 P45589 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 329357 T1495 C258 P51741 oasc.SolrCore.execute [collection1] webapp= path=/replication params={version=2&command=filelist&qt=/replication&wt=javabin&generation=3} status=0 QTime=0
[junit4:junit4] 2> 329358 T1538 C259 P45589 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 329358 T1538 C259 P45589 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index.20130622025830874
[junit4:junit4] 2> 329359 T1538 C259 P45589 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index.20130622025830874 lockFactory=org.apache.lucene.store.NativeFSLockFactory@42563224 fullCopy=false
[junit4:junit4] 2> 329361 T1497 C258 P51741 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_3&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=3} status=0 QTime=0
[junit4:junit4] 2> 329362 T1538 C259 P45589 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 329364 T1538 C259 P45589 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 329364 T1538 C259 P45589 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 329366 T1538 C259 P45589 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> 329366 T1538 C259 P45589 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 329367 T1538 C259 P45589 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 329367 T1538 C259 P45589 oass.SolrIndexSearcher.<init> Opening Searcher@e2f02b7 main
[junit4:junit4] 2> 329368 T1537 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e2f02b7 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 329368 T1538 C259 P45589 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index.20130622025830874 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index.20130622025830874;done=true>>]
[junit4:junit4] 2> 329368 T1538 C259 P45589 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index.20130622025830874
[junit4:junit4] 2> 329368 T1538 C259 P45589 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index.20130622025830874
[junit4:junit4] 2> 329368 T1538 C259 P45589 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 329369 T1538 C259 P45589 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 329369 T1538 C259 P45589 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 329369 T1538 C259 P45589 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 329370 T1538 C259 P45589 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 329522 T1555 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371884307847/collection1
[junit4:junit4] 2> 329522 T1555 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 329523 T1555 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 329523 T1555 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 329524 T1555 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371884307847/collection1/'
[junit4:junit4] 2> 329525 T1555 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1371884307847/collection1/lib/README' to classloader
[junit4:junit4] 2> 329525 T1555 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1371884307847/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 329550 T1555 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 329578 T1555 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 329579 T1555 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 329582 T1555 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 329874 T1555 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 329879 T1555 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 329881 T1555 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 329893 T1555 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 329896 T1555 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 329898 T1555 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 329899 T1555 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 329899 T1555 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 329899 T1555 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 329900 T1555 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 329900 T1555 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 329900 T1555 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 329900 T1555 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371884307847/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/
[junit4:junit4] 2> 329900 T1555 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4e604ca9
[junit4:junit4] 2> 329901 T1555 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 329901 T1555 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4
[junit4:junit4] 2> 329901 T1555 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index/
[junit4:junit4] 2> 329901 T1555 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 329902 T1555 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index
[junit4:junit4] 2> 329923 T1555 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 329923 T1555 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 329925 T1555 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 329925 T1555 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 329925 T1555 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 329926 T1555 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 329926 T1555 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 329926 T1555 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 329928 T1555 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 329928 T1555 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 329928 T1555 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 329930 T1555 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 329932 T1555 oass.SolrIndexSearcher.<init> Opening Searcher@2faa0280 main
[junit4:junit4] 2> 329933 T1555 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 329933 T1555 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 329937 T1558 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2faa0280 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 329939 T1555 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 329939 T1555 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59727 collection:collection1 shard:shard1
[junit4:junit4] 2> 329943 T1555 oasc.ZkController.register We are http://127.0.0.1:59727/collection1/ and leader is http://127.0.0.1:51741/collection1/
[junit4:junit4] 2> 329943 T1555 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59727
[junit4:junit4] 2> 329943 T1555 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 329943 T1555 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C260 name=collection1 org.apache.solr.core.SolrCore@66a1cf3d url=http://127.0.0.1:59727/collection1 node=127.0.0.1:59727_ C260_STATE=coll:collection1 core:collection1 props:{collection=collection1, base_url=http://127.0.0.1:59727, node_name=127.0.0.1:59727_, state=down, core=collection1}
[junit4:junit4] 2> 329943 T1559 C260 P59727 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 329944 T1559 C260 P59727 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 329944 T1555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 329945 T1559 C260 P59727 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 329945 T1559 C260 P59727 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 329946 T1460 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 329946 T1559 C260 P59727 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 329946 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 329946 T1460 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 329949 T1460 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 329949 T1495 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 329950 T1460 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 329950 T1460 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 329950 T1460 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 330168 T1483 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 330169 T1483 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:45589",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:45589_",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 330173 T1483 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59727",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:59727_",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 330177 T1552 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 330177 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 330177 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 330177 T1518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 330177 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 330177 T1489 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 330950 T1495 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 330950 T1495 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={action=PREPRECOVERY&version=2&checkLive=true&wt=javabin&nodeName=127.0.0.1:59727_&state=recovering&core=collection1&onlyIfLeader=true&coreNodeName=5} status=0 QTime=1001
[junit4:junit4] 2> 330951 T1460 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 331953 T1460 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C260_STATE=coll:collection1 core:collection1 props:{collection=collection1, base_url=http://127.0.0.1:59727, node_name=127.0.0.1:59727_, state=recovering, shard=shard1, core=collection1}
[junit4:junit4] 2> 332951 T1559 C260 P59727 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:51741/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 332951 T1559 C260 P59727 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59727 START replicas=[http://127.0.0.1:51741/collection1/] nUpdates=100
[junit4:junit4] 2> 332951 T1559 C260 P59727 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 332952 T1559 C260 P59727 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 332952 T1559 C260 P59727 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 332952 T1559 C260 P59727 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 332952 T1559 C260 P59727 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 332952 T1559 C260 P59727 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:51741/collection1/. core=collection1
[junit4:junit4] 2> 332952 T1559 C260 P59727 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C261 name=collection1 org.apache.solr.core.SolrCore@51713d58 url=http://127.0.0.1:51741/collection1 node=127.0.0.1:51741_ C261_STATE=coll:collection1 core:collection1 props:{collection=collection1, base_url=http://127.0.0.1:51741, node_name=127.0.0.1:51741_, state=active, shard=shard1, core=collection1, leader=true}
[junit4:junit4] 2> 332953 T1496 C261 P51741 oasc.SolrCore.execute [collection1] webapp= path=/get params={version=2&distrib=false&getVersions=100&wt=javabin&qt=/get} status=0 QTime=0
[junit4:junit4] 2> 332955 T1460 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 332957 T1498 C261 P51741 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 332961 T1498 C261 P51741 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> 332962 T1498 C261 P51741 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 332962 T1498 C261 P51741 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 332962 T1498 C261 P51741 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&commit_end_point=true&waitSearcher=true&wt=javabin&openSearcher=false&commit=true&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 332963 T1559 C260 P59727 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 332963 T1559 C260 P59727 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 332965 T1497 C261 P51741 oasc.SolrCore.execute [collection1] webapp= path=/replication params={version=2&command=indexversion&qt=/replication&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 332966 T1559 C260 P59727 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4] 2> 332966 T1559 C260 P59727 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 332966 T1559 C260 P59727 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 332968 T1498 C261 P51741 oasc.SolrCore.execute [collection1] webapp= path=/replication params={version=2&command=filelist&qt=/replication&wt=javabin&generation=4} status=0 QTime=0
[junit4:junit4] 2> 332969 T1559 C260 P59727 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 332970 T1559 C260 P59727 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index.20130622025834485
[junit4:junit4] 2> 332970 T1559 C260 P59727 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index.20130622025834485 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3ad0acca fullCopy=false
[junit4:junit4] 2> 332973 T1495 C261 P51741 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_4&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=4} status=0 QTime=0
[junit4:junit4] 2> 332975 T1559 C260 P59727 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 332976 T1559 C260 P59727 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 332976 T1559 C260 P59727 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 332979 T1559 C260 P59727 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> 332979 T1559 C260 P59727 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 332979 T1559 C260 P59727 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 332980 T1559 C260 P59727 oass.SolrIndexSearcher.<init> Opening Searcher@9abf38c main
[junit4:junit4] 2> 332980 T1558 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9abf38c main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 332981 T1559 C260 P59727 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index.20130622025834485 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index.20130622025834485;done=true>>]
[junit4:junit4] 2> 332981 T1559 C260 P59727 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index.20130622025834485
[junit4:junit4] 2> 332981 T1559 C260 P59727 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index.20130622025834485
[junit4:junit4] 2> 332981 T1559 C260 P59727 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 332981 T1559 C260 P59727 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 332981 T1559 C260 P59727 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 332981 T1559 C260 P59727 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 332982 T1559 C260 P59727 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 333184 T1483 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 333184 T1483 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59727",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:59727_",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 333202 T1552 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 333202 T1489 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 333202 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 333202 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 333202 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 333202 T1518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 333956 T1460 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 333957 T1460 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C262 name=collection1 org.apache.solr.core.SolrCore@368aaac4 url=http://127.0.0.1:55891/collection1 node=127.0.0.1:55891_ C262_STATE=coll:control_collection core:collection1 props:{collection=control_collection, base_url=http://127.0.0.1:55891, node_name=127.0.0.1:55891_, state=active, shard=shard1, core=collection1, leader=true}
[junit4:junit4] 2> 333962 T1475 C262 P55891 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 333965 T1475 C262 P55891 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 333965 T1475 C262 P55891 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 333971 T1475 C262 P55891 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/control/data/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 333972 T1475 C262 P55891 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 333972 T1475 C262 P55891 oass.SolrIndexSearcher.<init> Opening Searcher@7e3ba5d0 main
[junit4:junit4] 2> 333972 T1475 C262 P55891 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 333973 T1486 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7e3ba5d0 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 333973 T1475 C262 P55891 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 11
[junit4:junit4] 2> 333976 T1495 C261 P51741 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:51741/collection1/, StdNode: http://127.0.0.1:59479/collection1/, StdNode: http://127.0.0.1:45589/collection1/, StdNode: http://127.0.0.1:59727/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 333979 T1496 C261 P51741 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> C260_STATE=coll:collection1 core:collection1 props:{collection=collection1, base_url=http://127.0.0.1:59727, node_name=127.0.0.1:59727_, state=active, shard=shard1, core=collection1}
[junit4:junit4] 2> 333983 T1545 C260 P59727 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C263 name=collection1 org.apache.solr.core.SolrCore@2a6509a2 url=http://127.0.0.1:45589/collection1 node=127.0.0.1:45589_ C263_STATE=coll:collection1 core:collection1 props:{collection=collection1, base_url=http://127.0.0.1:45589, node_name=127.0.0.1:45589_, state=active, shard=shard1, core=collection1}
[junit4:junit4] 2> 333986 T1531 C263 P45589 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C264 name=collection1 org.apache.solr.core.SolrCore@71243dc0 url=http://127.0.0.1:59479/collection1 node=127.0.0.1:59479_ C264_STATE=coll:collection1 core:collection1 props:{collection=collection1, base_url=http://127.0.0.1:59479, node_name=127.0.0.1:59479_, state=active, shard=shard1, core=collection1}
[junit4:junit4] 2> 333987 T1513 C264 P59479 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 334001 T1496 C261 P51741 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=4
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> 334002 T1496 C261 P51741 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 334002 T1496 C261 P51741 oass.SolrIndexSearcher.<init> Opening Searcher@60d7c783 main
[junit4:junit4] 2> 334002 T1496 C261 P51741 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 334003 T1504 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@60d7c783 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 334004 T1496 C261 P51741 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&commit_end_point=true&expungeDeletes=false&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 25
[junit4:junit4] 2> 334031 T1513 C264 P59479 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> 334032 T1513 C264 P59479 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 334031 T1545 C260 P59727 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> 334032 T1545 C260 P59727 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 334033 T1513 C264 P59479 oass.SolrIndexSearcher.<init> Opening Searcher@22aea6c9 main
[junit4:junit4] 2> 334031 T1531 C263 P45589 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> 334033 T1531 C263 P45589 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 334033 T1545 C260 P59727 oass.SolrIndexSearcher.<init> Opening Searcher@6530f7b0 main
[junit4:junit4] 2> 334033 T1513 C264 P59479 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 334034 T1531 C263 P45589 oass.SolrIndexSearcher.<init> Opening Searcher@7714c027 main
[junit4:junit4] 2> 334034 T1545 C260 P59727 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 334034 T1520 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@22aea6c9 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 334034 T1558 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6530f7b0 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 334034 T1531 C263 P45589 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 334035 T1513 C264 P59479 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&commit_end_point=true&expungeDeletes=false&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 48
[junit4:junit4] 2> 334035 T1537 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7714c027 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 334035 T1545 C260 P59727 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&commit_end_point=true&expungeDeletes=false&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 52
[junit4:junit4] 2> 334036 T1531 C263 P45589 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&commit_end_point=true&expungeDeletes=false&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 50
[junit4:junit4] 2> 334036 T1495 C261 P51741 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 60
[junit4:junit4] 2> 334037 T1460 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 334039 T1497 C261 P51741 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&distrib=false&rows=0&q=*:*&wt=javabin&tests=checkShardConsistency} hits=0 status=0 QTime=0
[junit4:junit4] 2> 334041 T1512 C264 P59479 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&distrib=false&rows=0&q=*:*&wt=javabin&tests=checkShardConsistency} hits=0 status=0 QTime=1
[junit4:junit4] 2> 334042 T1528 C263 P45589 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&distrib=false&rows=0&q=*:*&wt=javabin&tests=checkShardConsistency} hits=0 status=0 QTime=0
[junit4:junit4] 2> 334044 T1547 C260 P59727 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&distrib=false&rows=0&q=*:*&wt=javabin&tests=checkShardConsistency} hits=0 status=0 QTime=0
[junit4:junit4] 2> 336049 T1478 C262 P55891 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {deleteByQuery=*:* (-1438524970172940288)} 0 2
[junit4:junit4] 2> 336056 T1531 C263 P45589 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&_version_=-1438524970178183168&update.from=http://127.0.0.1:51741/collection1/} {deleteByQuery=*:* (-1438524970178183168)} 0 1
[junit4:junit4] 2> 336056 T1514 C264 P59479 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&_version_=-1438524970178183168&update.from=http://127.0.0.1:51741/collection1/} {deleteByQuery=*:* (-1438524970178183168)} 0 1
[junit4:junit4] 2> 336056 T1545 C260 P59727 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&_version_=-1438524970178183168&update.from=http://127.0.0.1:51741/collection1/} {deleteByQuery=*:* (-1438524970178183168)} 0 1
[junit4:junit4] 2> 336057 T1497 C261 P51741 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=TOLEADER} {deleteByQuery=*:* (-1438524970178183168)} 0 5
[junit4:junit4] 2> 336058 T1548 C260 P59727 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {deleteByQuery=*:*} 0 8
[junit4:junit4] 2> 336062 T1476 C262 P55891 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[0 (1438524970186571776)]} 0 2
[junit4:junit4] 2> 336070 T1546 C260 P59727 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51741/collection1/} {add=[0 (1438524970192863232)]} 0 1
[junit4:junit4] 2> 336070 T1513 C264 P59479 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51741/collection1/} {add=[0 (1438524970192863232)]} 0 1
[junit4:junit4] 2> 336070 T1529 C263 P45589 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51741/collection1/} {add=[0 (1438524970192863232)]} 0 1
[junit4:junit4] 2> 336071 T1498 C261 P51741 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=TOLEADER&distrib.from=http://127.0.0.1:59479/collection1/} {add=[0 (1438524970192863232)]} 0 6
[junit4:junit4] 2> 336072 T1512 C264 P59479 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[0]} 0 9
[junit4:junit4] 2> 336075 T1477 C262 P55891 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[1 (1438524970201251840)]} 0 1
[junit4:junit4] 2> 336081 T1511 C264 P59479 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51741/collection1/} {add=[1 (1438524970204397568)]} 0 1
[junit4:junit4] 2> 336081 T1530 C263 P45589 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51741/collection1/} {add=[1 (1438524970204397568)]} 0 1
[junit4:junit4] 2> 336081 T1547 C260 P59727 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51741/collection1/} {add=[1 (1438524970204397568)]} 0 1
[junit4:junit4] 2> 336082 T1496 C261 P51741 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[1 (1438524970204397568)]} 0 5
[junit4:junit4] 2> 336084 T1475 C262 P55891 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[2 (1438524970211737600)]} 0 0
[junit4:junit4] 2> 336088 T1495 C261 P51741 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:59479/collection1/]
[junit4:junit4] 2> 336088 T1495 C261 P51741 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:59479/collection1/ against:[http://127.0.0.1:59479/collection1/] result:true
[junit4:junit4] 2> 336088 T1495 C261 P51741 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:45589/collection1/ against:[http://127.0.0.1:59479/collection1/] result:false
[junit4:junit4] 2> 336089 T1495 C261 P51741 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:59727/collection1/ against:[http://127.0.0.1:59479/collection1/] result:false
[junit4:junit4] 2> 336093 T1531 C263 P45589 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:59479/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51741/collection1/} {add=[2 (1438524970216980480)]} 0 1
[junit4:junit4] 2> 336093 T1545 C260 P59727 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:59479/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51741/collection1/} {add=[2 (1438524970216980480)]} 0 1
[junit4:junit4] 2> 336094 T1495 C261 P51741 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:59479/collection1/&version=2&wt=javabin&update.distrib=TOLEADER&distrib.from=http://127.0.0.1:45589/collection1/} {add=[2 (1438524970216980480)]} 0 6
[junit4:junit4] 2> 336095 T1528 C263 P45589 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:59479/collection1/&version=2&wt=javabin} {add=[2]} 0 9
[junit4:junit4] 2> 336098 T1478 C262 P55891 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[3 (1438524970225369088)]} 0 1
[junit4:junit4] 2> 336102 T1497 C261 P51741 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:59479/collection1/]
[junit4:junit4] 2> 336103 T1497 C261 P51741 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:59479/collection1/ against:[http://127.0.0.1:59479/collection1/] result:true
[junit4:junit4] 2> 336103 T1497 C261 P51741 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:45589/collection1/ against:[http://127.0.0.1:59479/collection1/] result:false
[junit4:junit4] 2> 336103 T1497 C261 P51741 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:59727/collection1/ against:[http://127.0.0.1:59479/collection1/] result:false
[junit4:junit4] 2> 336107 T1529 C263 P45589 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:59479/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51741/collection1/} {add=[3 (1438524970231660544)]} 0 1
[junit4:junit4] 2> 336107 T1548 C260 P59727 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:59479/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51741/collection1/} {add=[3 (1438524970231660544)]} 0 1
[junit4:junit4] 2> 336108 T1497 C261 P51741 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:59479/collection1/&version=2&wt=javabin&update.distrib=TOLEADER&distrib.from=http://127.0.0.1:59479/collection1/} {add=[3 (1438524970231660544)]} 0 6
[junit4:junit4] 2> 336108 T1514 C264 P59479 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:59479/collection1/&test.distrib.skip.servers=http://127.0.0.1:45589/collection1/&version=2&wt=javabin} {add=[3]} 0 8
[junit4:junit4] 2> 336110 T1476 C262 P55891 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 336131 T1476 C262 P55891 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/control/data/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/control/data/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> 336132 T1476 C262 P55891 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 336135 T1476 C262 P55891 oass.SolrIndexSearcher.<init> Opening Searcher@4734060f main
[junit4:junit4] 2> 336135 T1476 C262 P55891 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 336135 T1486 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4734060f main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 336136 T1476 C262 P55891 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 26
[junit4:junit4] 2> 336138 T1513 C264 P59479 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:51741/collection1/, StdNode: http://127.0.0.1:59479/collection1/, StdNode: http://127.0.0.1:45589/collection1/, StdNode: http://127.0.0.1:59727/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 336140 T1498 C261 P51741 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 336141 T1530 C263 P45589 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 336141 T1511 C264 P59479 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 336141 T1546 C260 P59727 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 336181 T1530 C263 P45589 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> 336182 T1530 C263 P45589 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 336181 T1498 C261 P51741 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_6,generation=6}
[junit4:junit4] 2> 336182 T1498 C261 P51741 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 336181 T1511 C264 P59479 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> 336183 T1511 C264 P59479 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 336185 T1530 C263 P45589 oass.SolrIndexSearcher.<init> Opening Searcher@39410485 main
[junit4:junit4] 2> 336185 T1530 C263 P45589 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 336186 T1537 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@39410485 main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 336186 T1498 C261 P51741 oass.SolrIndexSearcher.<init> Opening Searcher@609f0ce main
[junit4:junit4] 2> 336186 T1530 C263 P45589 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&commit_end_point=true&expungeDeletes=false&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 45
[junit4:junit4] 2> 336186 T1498 C261 P51741 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 336186 T1511 C264 P59479 oass.SolrIndexSearcher.<init> Opening Searcher@6768254 main
[junit4:junit4] 2> 336187 T1511 C264 P59479 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 336187 T1504 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@609f0ce main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 336187 T1520 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6768254 main{StandardDirectoryReader(segments_4:3:nrt _0(5.0):C2)}
[junit4:junit4] 2> 336187 T1498 C261 P51741 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&commit_end_point=true&expungeDeletes=false&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 47
[junit4:junit4] 2> 336187 T1511 C264 P59479 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&commit_end_point=true&expungeDeletes=false&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 46
[junit4:junit4] 2> 336191 T1546 C260 P59727 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index,segFN=segments_6,generation=6}
[junit4:junit4] 2> 336192 T1546 C260 P59727 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 336194 T1546 C260 P59727 oass.SolrIndexSearcher.<init> Opening Searcher@1bf48a87 main
[junit4:junit4] 2> 336194 T1546 C260 P59727 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 336195 T1558 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1bf48a87 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 336195 T1546 C260 P59727 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&commit_end_point=true&expungeDeletes=false&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 54
[junit4:junit4] 2> 336196 T1513 C264 P59479 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 58
[junit4:junit4] 2> 336196 T1460 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 336197 T1460 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 336197 T1460 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 336200 T1496 C261 P51741 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&distrib=false&rows=0&q=*:*&wt=javabin&tests=checkShardConsistency} hits=4 status=0 QTime=1
[junit4:junit4] 2> 336202 T1514 C264 P59479 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&distrib=false&rows=0&q=*:*&wt=javabin&tests=checkShardConsistency} hits=2 status=0 QTime=0
[junit4:junit4] 2> 336204 T1531 C263 P45589 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&distrib=false&rows=0&q=*:*&wt=javabin&tests=checkShardConsistency} hits=4 status=0 QTime=0
[junit4:junit4] 2> 336207 T1547 C260 P59727 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&distrib=false&rows=0&q=*:*&wt=javabin&tests=checkShardConsistency} hits=4 status=0 QTime=1
[junit4:junit4] 2> 336207 T1460 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 336211 T1528 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : action=SYNCSHARD&collection=collection1&version=2&wt=javabin&shard=shard1
[junit4:junit4] 2> 336212 T1528 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 336216 T1495 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4] 2> 336216 T1495 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51741/collection1/
[junit4:junit4] 2> 336217 T1495 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:51741 START replicas=[http://127.0.0.1:59479/collection1/, http://127.0.0.1:45589/collection1/, http://127.0.0.1:59727/collection1/] nUpdates=100
[junit4:junit4] 2> 336218 T1512 C264 P59479 oasc.SolrCore.execute [collection1] webapp= path=/get params={version=2&distrib=false&getVersions=100&wt=javabin&qt=/get} status=0 QTime=0
[junit4:junit4] 2> 336218 T1529 C263 P45589 oasc.SolrCore.execute [collection1] webapp= path=/get params={version=2&distrib=false&getVersions=100&wt=javabin&qt=/get} status=0 QTime=0
[junit4:junit4] 2> 336219 T1495 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:51741 Received 3 versions from 127.0.0.1:59479/collection1/
[junit4:junit4] 2> 336219 T1495 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:51741 Our versions are newer. ourLowThreshold=1438524970178183168 otherHigh=1438524970204397568
[junit4:junit4] 2> 336219 T1545 C260 P59727 oasc.SolrCore.execute [collection1] webapp= path=/get params={version=2&distrib=false&getVersions=100&wt=javabin&qt=/get} status=0 QTime=0
[junit4:junit4] 2> 336219 T1495 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:51741 Received 5 versions from 127.0.0.1:45589/collection1/
[junit4:junit4] 2> 336220 T1495 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:51741 Our versions are newer. ourLowThreshold=1438524970178183168 otherHigh=1438524970216980480
[junit4:junit4] 2> 336220 T1495 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:51741 Received 5 versions from 127.0.0.1:59727/collection1/
[junit4:junit4] 2> 336220 T1495 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:51741 Our versions are newer. ourLowThreshold=1438524970178183168 otherHigh=1438524970216980480
[junit4:junit4] 2> 336220 T1495 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:51741 DONE. sync succeeded
[junit4:junit4] 2> 336221 T1495 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 336221 T1495 oasc.SyncStrategy.syncToMe http://127.0.0.1:51741/collection1/: try and ask http://127.0.0.1:59479/collection1/ to sync
[junit4:junit4] 2> 336222 T1495 oasc.SyncStrategy.syncToMe http://127.0.0.1:51741/collection1/: try and ask http://127.0.0.1:45589/collection1/ to sync
[junit4:junit4] 2> 336222 T1495 oasc.SyncStrategy.syncToMe http://127.0.0.1:51741/collection1/: try and ask http://127.0.0.1:59727/collection1/ to sync
[junit4:junit4] 2> 336223 T1511 C264 P59479 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59479 START replicas=[http://127.0.0.1:51741/collection1/] nUpdates=100
[junit4:junit4] 2> 336223 T1530 C263 P45589 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45589 START replicas=[http://127.0.0.1:51741/collection1/] nUpdates=100
[junit4:junit4] 2> 336224 T1548 C260 P59727 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59727 START replicas=[http://127.0.0.1:51741/collection1/] nUpdates=100
[junit4:junit4] 2> 336224 T1497 C261 P51741 oasc.SolrCore.execute [collection1] webapp= path=/get params={version=2&distrib=false&getVersions=100&wt=javabin&qt=/get} status=0 QTime=0
[junit4:junit4] 2> 336225 T1498 C261 P51741 oasc.SolrCore.execute [collection1] webapp= path=/get params={version=2&distrib=false&getVersions=100&wt=javabin&qt=/get} status=0 QTime=0
[junit4:junit4] 2> 336225 T1530 C263 P45589 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45589 Received 5 versions from 127.0.0.1:51741/collection1/
[junit4:junit4] 2> 336225 T1511 C264 P59479 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59479 Received 5 versions from 127.0.0.1:51741/collection1/
[junit4:junit4] 2> 336225 T1530 C263 P45589 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45589 Our versions are newer. ourLowThreshold=1438524970178183168 otherHigh=1438524970216980480
[junit4:junit4] 2> 336226 T1496 C261 P51741 oasc.SolrCore.execute [collection1] webapp= path=/get params={version=2&distrib=false&getVersions=100&wt=javabin&qt=/get} status=0 QTime=0
[junit4:junit4] 2> 336226 T1511 C264 P59479 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:59479 Requesting updates from 127.0.0.1:51741/collection1/n=2 versions=[1438524970231660544, 1438524970216980480]
[junit4:junit4] 2> 336226 T1548 C260 P59727 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59727 Received 5 versions from 127.0.0.1:51741/collection1/
[junit4:junit4] 2> 336226 T1530 C263 P45589 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45589 DONE. sync succeeded
[junit4:junit4] 2> 336226 T1548 C260 P59727 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:59727 Our versions are newer. ourLowThreshold=1438524970178183168 otherHigh=1438524970216980480
[junit4:junit4] 2> 336227 T1548 C260 P59727 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59727 DONE. sync succeeded
[junit4:junit4] 2> 336227 T1530 C263 P45589 oasc.SolrCore.execute [collection1] webapp= path=/get params={version=2&distrib=false&getVersions=100&wt=javabin&qt=/get&sync=http://127.0.0.1:51741/collection1/} status=0 QTime=4
[junit4:junit4] 2> 336227 T1548 C260 P59727 oasc.SolrCore.execute [collection1] webapp= path=/get params={version=2&distrib=false&getVersions=100&wt=javabin&qt=/get&sync=http://127.0.0.1:51741/collection1/} status=0 QTime=3
[junit4:junit4] 2> 336228 T1495 oasc.SyncStrategy.syncToMe http://127.0.0.1:51741/collection1/: sync completed with http://127.0.0.1:45589/collection1/
[junit4:junit4] 2> 336228 T1495 oasc.SyncStrategy.syncToMe http://127.0.0.1:51741/collection1/: sync completed with http://127.0.0.1:59727/collection1/
[junit4:junit4] 2> 336228 T1497 C261 P51741 oasc.SolrCore.execute [collection1] webapp= path=/get params={version=2&distrib=false&getUpdates=1438524970231660544,1438524970216980480&wt=javabin&qt=/get} status=0 QTime=0
[junit4:junit4] 2> 336231 T1511 C264 P59479 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1438524970216980480), 3 (1438524970231660544)]} 0 3
[junit4:junit4] 2> 336231 T1511 C264 P59479 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59479 DONE. sync succeeded
[junit4:junit4] 2> 336231 T1511 C264 P59479 oasc.SolrCore.execute [collection1] webapp= path=/get params={version=2&distrib=false&getVersions=100&wt=javabin&qt=/get&sync=http://127.0.0.1:51741/collection1/} status=0 QTime=8
[junit4:junit4] 2> 336232 T1495 oasc.SyncStrategy.syncToMe http://127.0.0.1:51741/collection1/: sync completed with http://127.0.0.1:59479/collection1/
[junit4:junit4] 2> 336232 T1495 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={action=REQUESTSYNCSHARD&collection=collection1&version=2&wt=javabin&core=collection1&shard=shard1} status=0 QTime=16
[junit4:junit4] 2> 336233 T1528 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={action=SYNCSHARD&collection=collection1&version=2&wt=javabin&shard=shard1} status=0 QTime=22
[junit4:junit4] 2> 336233 T1460 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 336233 T1460 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 336234 T1460 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 336234 T1460 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 336236 T1477 C262 P55891 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 336281 T1477 C262 P55891 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/control/data/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/control/data/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> 336282 T1477 C262 P55891 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 336282 T1477 C262 P55891 oass.SolrIndexSearcher.<init> Opening Searcher@6d9a1155 main
[junit4:junit4] 2> 336283 T1477 C262 P55891 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 336283 T1486 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6d9a1155 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 336283 T1477 C262 P55891 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 47
[junit4:junit4] 2> 336286 T1513 C264 P59479 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:51741/collection1/, StdNode: http://127.0.0.1:59479/collection1/, StdNode: http://127.0.0.1:45589/collection1/, StdNode: http://127.0.0.1:59727/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 336288 T1514 C264 P59479 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 336288 T1531 C263 P45589 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 336288 T1498 C261 P51741 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 336288 T1546 C260 P59727 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 336311 T1531 C263 P45589 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty3/index,segFN=segments_6,generation=6}
[junit4:junit4] 2> 336312 T1531 C263 P45589 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 336312 T1531 C263 P45589 oass.SolrIndexSearcher.<init> Opening Searcher@36e13e0 main
[junit4:junit4] 2> 336312 T1531 C263 P45589 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 336313 T1537 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@36e13e0 main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 336314 T1531 C263 P45589 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&commit_end_point=true&expungeDeletes=false&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 26
[junit4:junit4] 2> 336331 T1498 C261 P51741 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_6,generation=6}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty1/index,segFN=segments_7,generation=7}
[junit4:junit4] 2> 336332 T1546 C260 P59727 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index,segFN=segments_6,generation=6}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty4/index,segFN=segments_7,generation=7}
[junit4:junit4] 2> 336332 T1498 C261 P51741 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 7
[junit4:junit4] 2> 336332 T1546 C260 P59727 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 7
[junit4:junit4] 2> 336333 T1498 C261 P51741 oass.SolrIndexSearcher.<init> Opening Searcher@297df329 main
[junit4:junit4] 2> 336333 T1498 C261 P51741 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 336333 T1546 C260 P59727 oass.SolrIndexSearcher.<init> Opening Searcher@55445b9d main
[junit4:junit4] 2> 336334 T1546 C260 P59727 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 336334 T1504 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@297df329 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 336334 T1558 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@55445b9d main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 336334 T1498 C261 P51741 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&commit_end_point=true&expungeDeletes=false&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 46
[junit4:junit4] 2> 336334 T1546 C260 P59727 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&commit_end_point=true&expungeDeletes=false&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 46
[junit4:junit4] 2> 336371 T1514 C264 P59479 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371884297708/jetty2/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> 336372 T1514 C264 P59479 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 336374 T1514 C264 P59479 oass.SolrIndexSearcher.<init> Opening Searcher@5341d6fa main
[junit4:junit4] 2> 336375 T1514 C264 P59479 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 336375 T1520 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5341d6fa main{StandardDirectoryReader(segments_5:5:nrt _0(5.0):C2 _1(5.0):C2)}
[junit4:junit4] 2> 336375 T1514 C264 P59479 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&commit_end_point=true&expungeDeletes=false&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 87
[junit4:junit4] 2> 336376 T1513 C264 P59479 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&waitSearcher=true&wt=javabin&commit=true&softCommit=false} {commit=} 0 90
[junit4:junit4] 2> 336376 T1460 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 336379 T1496 C261 P51741 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&distrib=false&rows=0&q=*:*&wt=javabin&tests=checkShardConsistency} hits=4 status=0 QTime=0
[junit4:junit4] 2> 336381 T1512 C264 P59479 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&distrib=false&rows=0&q=*:*&wt=javabin&tests=checkShardConsistency} hits=4 status=0 QTime=0
[junit4:junit4] 2> 336383 T1529 C263 P45589 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&distrib=false&rows=0&q=*:*&wt=javabin&tests=checkShardConsistency} hits=4 status=0 QTime=0
[junit4:junit4] 2> 336386 T1547 C260 P59727 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&distrib=false&rows=0&q=*:*&wt=javabin&tests=checkShardConsistency} hits=4 status=0 QTime=1
[junit4:junit4] 2> 338433 T1460 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> check const of shard1
[junit4:junit4] 2> client0
[junit4:junit4] 2> PROPS:2:{"collection":"collection1","base_url":"http://127.0.0.1:51741","node_name":"127.0.0.1:51741_","state":"active","shard":"shard1","core":"collection1","leader":"true"}
[junit4:junit4] 2> 338770 T1496 C261 P51741 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&distrib=false&rows=0&q=*:*&wt=javabin&tests=checkShardConsistency} hits=4 status=0 QTime=108
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client1
[junit4:junit4] 2> PROPS:3:{"collection":"collection1","base_url":"http://127.0.0.1:59479","node_name":"127.0.0.1:59479_","state":"active","shard":"shard1","core":"collection1"}
[junit4:junit4] 2> 338838 T1514 C264 P59479 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&distrib=false&rows=0&q=*:*&wt=javabin&tests=checkShardConsistency} hits=4 status=0 QTime=5
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client2
[junit4:junit4] 2> PROPS:4:{"collection":"collection1","base_url":"http://127.0.0.1:45589","node_name":"127.0.0.1:45589_","state":"active","shard":"shard1","core":"collection1"}
[junit4:junit4] 2> 338864 T1530 C263 P45589 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&distrib=false&rows=0&q=*:*&wt=javabin&tests=checkShardConsistency} hits=4 status=0 QTime=7
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> client3
[junit4:junit4] 2> PROPS:5:{"collection":"collection1","base_url":"http://127.0.0.1:59727","node_name":"127.0.0.1:59727_","state":"active","shard":"shard1","core":"collection1"}
[junit4:junit4] 2> 338879 T1545 C260 P59727 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&distrib=false&rows=0&q=*:*&wt=javabin&tests=checkShardConsistency} hits=4 status=0 QTime=0
[junit4:junit4] 2> live:true
[junit4:junit4] 2> num:4
[junit4:junit4] 2>
[junit4:junit4] 2> 338884 T1513 C264 P59479 oasc.SolrCore.execute [collection1] webapp= path=/select params={version=2&q=*:*&wt=javabin} hits=4 status=0 QTime=0
[junit4:junit4] 2> 338974 T1478 C262 P55891 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={version=2&wt=javabin} {add=[4 (1438524973169770496)]} 0 75
[junit4:junit4] 2> 338979 T1498 C261 P51741 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:45589/collection1/]
[junit4:junit4] 2> 338980 T1498 C261 P51741 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:59479/collection1/ against:[http://127.0.0.1:45589/collection1/] result:false
[junit4:junit4] 2> 338980 T1498 C261 P51741 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:45589/collection1/ against:[http://127.0.0.1:45589/collection1/] result:true
[junit4:junit4] 2> 338981 T1498 C261 P51741 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:59727/collection1/ against:[http://127.0.0.1:45589/collection1/] result:false
[junit4:junit4] 2> 338994 T1512 C264 P59479 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:45589/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51741/collection1/} {add=[4 (1438524973249462272)]} 0 6
[junit4:junit4] 2> 338996 T1548 C260 P59727 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:45589/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51741/collection1/} {add=[4 (1438524973249462272)]} 0 7
[junit4:junit4] 2> 338997 T1498 C261 P51741 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:45589/collection1/&version=2&wt=javabin} {add=[4 (1438524973249462272)]} 0 19
[junit4:junit4] 2> 338998 T1460 oasc.ChaosMonkey.monkeyLog monkey: kill shard! 51741
[junit4:junit4] 2> 339036 T1460 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=156946197
[junit4:junit4] 2> 339213 T1483 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 339215 T1483 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51741",
[junit4:junit4] 2> "node_name":"127.0.0.1:51741_",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "roles":null}
[junit4:junit4] 2> 339219 T1552 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 339219 T1489 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 339219 T1502 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 339219 T1518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 339220 T1482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json,
[...truncated too long message...]
junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89907809334722563-1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:55891",
[junit4:junit4] 1> "node_name":"127.0.0.1:55891_",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89907809334722574-2-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89907809334722570-4-n_0000000005 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89907809334722572-5-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (1)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:59727",
[junit4:junit4] 1> "node_name":"127.0.0.1:59727_",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/configs (1)
[junit4:junit4] 1> /solr/configs/conf1 (9)
[junit4:junit4] 1> /solr/configs/conf1/open-exchange-rates.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "disclaimer": "This data is not real, it was synthetically created to match currency.xml. It is modeled after the data format available from openexchangerates.org. See https://openexchangerates.org/documentation for details",
[junit4:junit4] 1> "license": "http://www.apache.org/licenses/LICENSE-2.0",
[junit4:junit4] 1> "timestamp": 1332070464,
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 1> "IMPORTANT NOTE": "In order for tests to work, this data must be kept in sync with ./currency.xml",
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 1> "base": "USD",
[junit4:junit4] 1> "rates": {
[junit4:junit4] 1> "USD": 1,
[junit4:junit4] 1> "JPY": 81.29,
[junit4:junit4] 1> "EUR": 2.5,
[junit4:junit4] 1> "GBP": 0.5,
[junit4:junit4] 1> "MXN": 2.0
[junit4:junit4] 1> }
[junit4:junit4] 1> }
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/configs/conf1/stopwords.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/old_synonyms.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/solrconfig.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/mapping-ISOLatin1Accent.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/schema.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/currency.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/synonyms.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/protwords.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:51741_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:59727_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:45589_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:55891_ (0)
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=39BC0AE900E83EF -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_HN -Dtests.timezone=America/Manaus -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 99.8s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:51741/collection1lastClient and got 246 from http://127.0.0.1:45589/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([39BC0AE900E83EF:827D4EB6E751E3D3]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 416028 T1460 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 99844 T1459 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 417155 T1552 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 417156 T1552 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 417156 T1552 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=es_HN, timezone=America/Manaus
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.8.0-ea (64-bit)/cpus=8,threads=1,free=59449808,total=356450304
[junit4:junit4] 2> NOTE: All tests run in this JVM: [ReturnFieldsTest, WordBreakSolrSpellCheckerTest, SystemInfoHandlerTest, FieldMutatingUpdateProcessorTest, QueryParsingTest, SolrIndexSplitterTest, DateFieldTest, TestGroupingSearch, PreAnalyzedFieldTest, TestReversedWildcardFilterFactory, DistributedQueryElevationComponentTest, TestRemoteStreaming, ZkControllerTest, TestDocSet, JsonLoaderTest, TestSchemaResource, TestTrie, PrimUtilsTest, SolrCoreTest, TestPluginEnable, TestSolrXml, TestCharFilters, TestExtendedDismaxParser, RecoveryZkTest, PolyFieldTest, RAMDirectoryFactoryTest, TestSolrDeletionPolicy2, TestPropInjectDefaults, TestClassNameShortening, SearchHandlerTest, MBeansHandlerTest, TestConfig, TestFastLRUCache, RegexBoostProcessorTest, TermsComponentTest, ResponseLogComponentTest, TestIBSimilarityFactory, TestFastWriter, QueryElevationComponentTest, TestPHPSerializedResponseWriter, FastVectorHighlighterTest, TestDynamicFieldResource, SOLR749Test, TermVectorComponentTest, TestLRUCache, PreAnalyzedUpdateProcessorTest, TestCoreContainer, TestPhraseSuggestions, SolrInfoMBeanTest, TestXIncludeConfig, BinaryUpdateRequestHandlerTest, OverseerCollectionProcessorTest, DirectUpdateHandlerOptimizeTest, TestQueryUtils, BadComponentTest, SuggesterTest, DistributedSpellCheckComponentTest, ResourceLoaderTest, ShardRoutingTest, TestFuzzyAnalyzedSuggestions, TestSearchPerf, UpdateRequestProcessorFactoryTest, TestPerFieldSimilarity, UnloadDistributedZkTest, SolrTestCaseJ4Test, StatelessScriptUpdateProcessorFactoryTest, SynonymTokenizerTest, FileBasedSpellCheckerTest, ZkCLITest, DirectSolrConnectionTest, TestSolrCoreProperties, BasicDistributedZkTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, SyncSliceTest]
[junit4:junit4] Completed on J0 in 101.17s, 1 test, 1 failure <<< FAILURES!
[...truncated 447 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1253 tests, 1 failure, 16 ignored (7 assumptions)
Total time: 34 minutes 56 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-ea-b93 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_25) - Build #
6253 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6253/
Java: 64bit/jdk1.7.0_25 -XX:-UseCompressedOops -XX:+UseSerialGC
1 tests failed.
FAILED: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:46423/la_h/dt/collection1lastClient and got 257 from http://127.0.0.1:42901/la_h/dt/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:46423/la_h/dt/collection1lastClient and got 257 from http://127.0.0.1:42901/la_h/dt/collection1
at __randomizedtesting.SeedInfo.seed([1D2543F64D8FF15C:9CC3CDEE3AD09160]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9564 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4] 2> 840139 T2744 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /la_h/dt
[junit4:junit4] 2> 840146 T2744 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1371894634529
[junit4:junit4] 2> 840147 T2744 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 840148 T2745 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 840248 T2744 oasc.ZkTestServer.run start zk server on port:55758
[junit4:junit4] 2> 840249 T2744 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 840253 T2751 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30729af2 name:ZooKeeperConnection Watcher:127.0.0.1:55758 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 840253 T2744 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 840253 T2744 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 840257 T2744 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 840259 T2753 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33bde2f2 name:ZooKeeperConnection Watcher:127.0.0.1:55758/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 840259 T2744 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 840259 T2744 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 840264 T2744 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 840267 T2744 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 840269 T2744 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 840272 T2744 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 840273 T2744 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 840279 T2744 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 840280 T2744 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 840287 T2744 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 840287 T2744 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 840294 T2744 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 840295 T2744 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 840298 T2744 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 840299 T2744 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 840301 T2744 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 840302 T2744 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 840304 T2744 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 840305 T2744 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 840308 T2744 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 840309 T2744 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 840314 T2744 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 840315 T2744 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 840438 T2744 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 840456 T2744 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48059
[junit4:junit4] 2> 840457 T2744 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 840457 T2744 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 840458 T2744 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371894634703
[junit4:junit4] 2> 840458 T2744 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371894634703/solr.xml
[junit4:junit4] 2> 840458 T2744 oasc.CoreContainer.<init> New CoreContainer 612078369
[junit4:junit4] 2> 840459 T2744 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371894634703/'
[junit4:junit4] 2> 840459 T2744 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371894634703/'
[junit4:junit4] 2> 840505 T2744 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 840505 T2744 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 840506 T2744 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 840506 T2744 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 840506 T2744 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 840506 T2744 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 840507 T2744 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 840507 T2744 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 840507 T2744 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 840507 T2744 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 840510 T2744 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 840510 T2744 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55758/solr
[junit4:junit4] 2> 840511 T2744 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 840511 T2744 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 840513 T2764 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f44dc1a name:ZooKeeperConnection Watcher:127.0.0.1:55758 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 840513 T2744 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 840514 T2744 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 840516 T2744 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 840518 T2766 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63cc00ce name:ZooKeeperConnection Watcher:127.0.0.1:55758/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 840518 T2744 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 840520 T2744 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 840522 T2744 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 840524 T2744 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 840525 T2744 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48059_la_h%2Fdt
[junit4:junit4] 2> 840526 T2744 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48059_la_h%2Fdt
[junit4:junit4] 2> 840528 T2744 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 840531 T2744 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 840532 T2744 oasc.Overseer.start Overseer (id=89908486768754691-127.0.0.1:48059_la_h%2Fdt-n_0000000000) starting
[junit4:junit4] 2> 840535 T2744 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 840537 T2768 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 840537 T2744 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 840539 T2744 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 840540 T2744 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 840542 T2767 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 840544 T2769 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 840544 T2769 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 842045 T2767 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 842046 T2767 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:48059_la_h%2Fdt",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48059/la_h/dt"}
[junit4:junit4] 2> 842046 T2767 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 842046 T2767 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 842049 T2766 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 842545 T2769 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371894634703/collection1
[junit4:junit4] 2> 842545 T2769 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 842546 T2769 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 842546 T2769 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 842547 T2769 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371894634703/collection1/'
[junit4:junit4] 2> 842548 T2769 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1371894634703/collection1/lib/README' to classloader
[junit4:junit4] 2> 842548 T2769 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1371894634703/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 842570 T2769 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 842604 T2769 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 842605 T2769 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 842609 T2769 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 842943 T2769 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 842947 T2769 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 842950 T2769 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 842961 T2769 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 842964 T2769 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 842966 T2769 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 842967 T2769 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 842967 T2769 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 842967 T2769 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 842968 T2769 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 842968 T2769 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 842968 T2769 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 842969 T2769 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371894634703/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371894634529/control/data/
[junit4:junit4] 2> 842969 T2769 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@75baec8e
[junit4:junit4] 2> 842969 T2769 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 842970 T2769 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/control/data
[junit4:junit4] 2> 842970 T2769 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371894634529/control/data/index/
[junit4:junit4] 2> 842970 T2769 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371894634529/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 842970 T2769 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/control/data/index
[junit4:junit4] 2> 842973 T2769 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@415b13b9; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 842974 T2769 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 842975 T2769 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 842975 T2769 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 842976 T2769 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 842976 T2769 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 842976 T2769 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 842976 T2769 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 842977 T2769 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 842977 T2769 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 842977 T2769 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 842979 T2769 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 842982 T2769 oass.SolrIndexSearcher.<init> Opening Searcher@54951a0d main
[junit4:junit4] 2> 842983 T2769 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 842983 T2769 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 842987 T2770 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@54951a0d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 842988 T2769 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 842988 T2769 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48059/la_h/dt collection:control_collection shard:shard1
[junit4:junit4] 2> 842989 T2769 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 842997 T2769 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 843000 T2769 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 843000 T2769 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 843000 T2769 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48059/la_h/dt/collection1/
[junit4:junit4] 2> 843000 T2769 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 843000 T2769 oasc.SyncStrategy.syncToMe http://127.0.0.1:48059/la_h/dt/collection1/ has no replicas
[junit4:junit4] 2> 843001 T2769 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48059/la_h/dt/collection1/
[junit4:junit4] 2> 843001 T2769 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 843554 T2767 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 843561 T2766 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 843607 T2769 oasc.ZkController.register We are http://127.0.0.1:48059/la_h/dt/collection1/ and leader is http://127.0.0.1:48059/la_h/dt/collection1/
[junit4:junit4] 2> 843607 T2769 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48059/la_h/dt
[junit4:junit4] 2> 843607 T2769 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 843607 T2769 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 843607 T2769 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 843609 T2769 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 843610 T2744 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 843611 T2744 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 843611 T2744 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 843613 T2744 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 843614 T2744 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 843616 T2773 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c612345 name:ZooKeeperConnection Watcher:127.0.0.1:55758/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 843616 T2744 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 843617 T2744 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 843619 T2744 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 843678 T2744 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 843686 T2744 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46423
[junit4:junit4] 2> 843687 T2744 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 843687 T2744 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 843687 T2744 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371894638003
[junit4:junit4] 2> 843688 T2744 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371894638003/solr.xml
[junit4:junit4] 2> 843688 T2744 oasc.CoreContainer.<init> New CoreContainer 292268113
[junit4:junit4] 2> 843688 T2744 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371894638003/'
[junit4:junit4] 2> 843688 T2744 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371894638003/'
[junit4:junit4] 2> 843727 T2744 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 843728 T2744 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 843728 T2744 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 843728 T2744 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 843728 T2744 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 843729 T2744 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 843729 T2744 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 843729 T2744 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 843729 T2744 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 843730 T2744 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 843732 T2744 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 843733 T2744 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55758/solr
[junit4:junit4] 2> 843733 T2744 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 843733 T2744 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 843736 T2784 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a4bb22b name:ZooKeeperConnection Watcher:127.0.0.1:55758 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 843736 T2744 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 843738 T2744 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 843740 T2744 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 843742 T2786 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35eb6951 name:ZooKeeperConnection Watcher:127.0.0.1:55758/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 843742 T2744 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 843745 T2744 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 844747 T2744 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46423_la_h%2Fdt
[junit4:junit4] 2> 844749 T2744 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46423_la_h%2Fdt
[junit4:junit4] 2> 844752 T2773 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 844752 T2786 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 844752 T2766 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 844753 T2766 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 844756 T2787 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 844757 T2787 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 845065 T2767 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 845066 T2767 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:48059_la_h%2Fdt",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48059/la_h/dt"}
[junit4:junit4] 2> 845069 T2767 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46423_la_h%2Fdt",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46423/la_h/dt"}
[junit4:junit4] 2> 845069 T2767 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 845069 T2767 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 845072 T2773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 845072 T2786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 845072 T2766 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 845758 T2787 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371894638003/collection1
[junit4:junit4] 2> 845758 T2787 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 845759 T2787 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 845759 T2787 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 845760 T2787 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371894638003/collection1/'
[junit4:junit4] 2> 845761 T2787 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1371894638003/collection1/lib/README' to classloader
[junit4:junit4] 2> 845761 T2787 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1371894638003/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 845788 T2787 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 845815 T2787 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 845817 T2787 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 845826 T2787 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 846136 T2787 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 846141 T2787 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 846143 T2787 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 846157 T2787 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 846160 T2787 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 846163 T2787 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 846164 T2787 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 846165 T2787 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 846165 T2787 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 846166 T2787 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 846167 T2787 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 846167 T2787 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 846167 T2787 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371894638003/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/
[junit4:junit4] 2> 846167 T2787 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@75baec8e
[junit4:junit4] 2> 846168 T2787 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 846168 T2787 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1
[junit4:junit4] 2> 846168 T2787 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index/
[junit4:junit4] 2> 846168 T2787 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 846169 T2787 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index
[junit4:junit4] 2> 846172 T2787 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 846172 T2787 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 846174 T2787 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 846174 T2787 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 846175 T2787 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 846175 T2787 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 846176 T2787 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 846176 T2787 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 846176 T2787 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 846177 T2787 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 846177 T2787 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 846180 T2787 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 846182 T2787 oass.SolrIndexSearcher.<init> Opening Searcher@588d8f50 main
[junit4:junit4] 2> 846183 T2787 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 846183 T2787 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 846187 T2788 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@588d8f50 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 846189 T2787 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 846190 T2787 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46423/la_h/dt collection:collection1 shard:shard1
[junit4:junit4] 2> 846191 T2787 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 846198 T2787 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 846200 T2787 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 846200 T2787 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 846201 T2787 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46423/la_h/dt/collection1/
[junit4:junit4] 2> 846201 T2787 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 846201 T2787 oasc.SyncStrategy.syncToMe http://127.0.0.1:46423/la_h/dt/collection1/ has no replicas
[junit4:junit4] 2> 846201 T2787 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46423/la_h/dt/collection1/
[junit4:junit4] 2> 846201 T2787 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 846578 T2767 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 846583 T2766 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 846583 T2786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 846583 T2773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 846609 T2787 oasc.ZkController.register We are http://127.0.0.1:46423/la_h/dt/collection1/ and leader is http://127.0.0.1:46423/la_h/dt/collection1/
[junit4:junit4] 2> 846609 T2787 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46423/la_h/dt
[junit4:junit4] 2> 846609 T2787 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 846609 T2787 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 846609 T2787 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 846611 T2787 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 846612 T2744 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 846613 T2744 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 846613 T2744 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 846673 T2744 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 846675 T2744 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52290
[junit4:junit4] 2> 846676 T2744 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 846676 T2744 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 846676 T2744 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371894640999
[junit4:junit4] 2> 846676 T2744 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371894640999/solr.xml
[junit4:junit4] 2> 846677 T2744 oasc.CoreContainer.<init> New CoreContainer 158913157
[junit4:junit4] 2> 846677 T2744 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371894640999/'
[junit4:junit4] 2> 846677 T2744 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371894640999/'
[junit4:junit4] 2> 846726 T2744 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 846726 T2744 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 846727 T2744 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 846727 T2744 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 846727 T2744 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 846727 T2744 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 846728 T2744 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 846728 T2744 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 846728 T2744 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 846729 T2744 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 846731 T2744 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 846732 T2744 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55758/solr
[junit4:junit4] 2> 846732 T2744 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 846733 T2744 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 846735 T2800 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75b525a3 name:ZooKeeperConnection Watcher:127.0.0.1:55758 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 846735 T2744 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 846737 T2744 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 846739 T2744 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 846741 T2802 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e0d40a6 name:ZooKeeperConnection Watcher:127.0.0.1:55758/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 846741 T2744 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 846744 T2744 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 847746 T2744 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52290_la_h%2Fdt
[junit4:junit4] 2> 847747 T2744 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52290_la_h%2Fdt
[junit4:junit4] 2> 847749 T2773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 847750 T2786 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 847750 T2766 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 847750 T2802 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 847750 T2766 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 847750 T2786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 847751 T2773 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 847755 T2803 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 847755 T2803 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 848087 T2767 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 848088 T2767 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46423_la_h%2Fdt",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46423/la_h/dt"}
[junit4:junit4] 2> 848090 T2767 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52290_la_h%2Fdt",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52290/la_h/dt"}
[junit4:junit4] 2> 848090 T2767 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 848090 T2767 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 848092 T2766 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 848092 T2802 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 848092 T2773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 848092 T2786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 848757 T2803 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371894640999/collection1
[junit4:junit4] 2> 848757 T2803 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 848758 T2803 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 848758 T2803 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 848759 T2803 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371894640999/collection1/'
[junit4:junit4] 2> 848760 T2803 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1371894640999/collection1/lib/README' to classloader
[junit4:junit4] 2> 848760 T2803 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1371894640999/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 848789 T2803 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 848816 T2803 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 848817 T2803 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 848821 T2803 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 849143 T2803 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 849148 T2803 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 849150 T2803 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 849161 T2803 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 849164 T2803 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 849167 T2803 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 849168 T2803 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 849168 T2803 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 849168 T2803 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 849169 T2803 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 849169 T2803 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 849169 T2803 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 849170 T2803 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371894640999/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2/
[junit4:junit4] 2> 849170 T2803 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@75baec8e
[junit4:junit4] 2> 849170 T2803 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 849171 T2803 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2
[junit4:junit4] 2> 849171 T2803 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2/index/
[junit4:junit4] 2> 849171 T2803 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 849171 T2803 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2/index
[junit4:junit4] 2> 849175 T2803 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17f4673c; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 849175 T2803 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 849177 T2803 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 849177 T2803 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 849177 T2803 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 849178 T2803 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 849178 T2803 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 849178 T2803 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 849179 T2803 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 849179 T2803 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 849179 T2803 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 849181 T2803 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 849184 T2803 oass.SolrIndexSearcher.<init> Opening Searcher@3837798f main
[junit4:junit4] 2> 849185 T2803 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 849185 T2803 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 849189 T2804 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3837798f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 849190 T2803 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 849190 T2803 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52290/la_h/dt collection:collection1 shard:shard1
[junit4:junit4] 2> 849193 T2803 oasc.ZkController.register We are http://127.0.0.1:52290/la_h/dt/collection1/ and leader is http://127.0.0.1:46423/la_h/dt/collection1/
[junit4:junit4] 2> 849193 T2803 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52290/la_h/dt
[junit4:junit4] 2> 849194 T2803 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 849194 T2803 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C3479 name=collection1 org.apache.solr.core.SolrCore@bebc2b url=http://127.0.0.1:52290/la_h/dt/collection1 node=127.0.0.1:52290_la_h%2Fdt C3479_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:52290_la_h%2Fdt, base_url=http://127.0.0.1:52290/la_h/dt}
[junit4:junit4] 2> 849194 T2805 C3479 P52290 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 849194 T2805 C3479 P52290 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 849195 T2803 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 849195 T2805 C3479 P52290 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 849195 T2805 C3479 P52290 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 849195 T2744 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 849196 T2744 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 849196 T2805 C3479 P52290 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 849196 T2744 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 849201 T2782 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 849255 T2744 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 849257 T2744 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42901
[junit4:junit4] 2> 849257 T2744 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 849258 T2744 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 849258 T2744 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371894643583
[junit4:junit4] 2> 849258 T2744 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371894643583/solr.xml
[junit4:junit4] 2> 849259 T2744 oasc.CoreContainer.<init> New CoreContainer 979543845
[junit4:junit4] 2> 849259 T2744 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371894643583/'
[junit4:junit4] 2> 849259 T2744 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371894643583/'
[junit4:junit4] 2> 849304 T2744 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 849304 T2744 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 849305 T2744 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 849305 T2744 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 849305 T2744 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 849305 T2744 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 849306 T2744 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 849306 T2744 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 849306 T2744 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 849306 T2744 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 849309 T2744 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 849309 T2744 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55758/solr
[junit4:junit4] 2> 849310 T2744 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 849310 T2744 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 849313 T2817 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38610b99 name:ZooKeeperConnection Watcher:127.0.0.1:55758 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 849313 T2744 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 849314 T2744 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 849316 T2744 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 849317 T2819 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7646d52c name:ZooKeeperConnection Watcher:127.0.0.1:55758/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 849318 T2744 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 849321 T2744 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 849597 T2767 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 849598 T2767 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52290_la_h%2Fdt",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52290/la_h/dt"}
[junit4:junit4] 2> 849602 T2773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 849602 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 849602 T2786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 849602 T2766 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 849602 T2802 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 850202 T2782 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 850202 T2782 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=3&state=recovering&nodeName=127.0.0.1:52290_la_h%252Fdt&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 850323 T2744 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42901_la_h%2Fdt
[junit4:junit4] 2> 850324 T2744 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42901_la_h%2Fdt
[junit4:junit4] 2> 850326 T2773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 850326 T2786 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 850326 T2802 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 850326 T2786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 850326 T2766 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 850327 T2819 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 850327 T2766 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 850327 T2773 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 850327 T2802 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 850327 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 850332 T2820 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 850332 T2820 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 851107 T2767 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 851107 T2767 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42901_la_h%2Fdt",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42901/la_h/dt"}
[junit4:junit4] 2> 851108 T2767 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 851108 T2767 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 851111 T2773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 851111 T2802 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 851111 T2786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 851111 T2766 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 851111 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 851333 T2820 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371894643583/collection1
[junit4:junit4] 2> 851333 T2820 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 851334 T2820 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 851334 T2820 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 851335 T2820 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371894643583/collection1/'
[junit4:junit4] 2> 851336 T2820 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1371894643583/collection1/lib/README' to classloader
[junit4:junit4] 2> 851336 T2820 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1371894643583/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 851358 T2820 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 851394 T2820 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 851396 T2820 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 851399 T2820 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 851732 T2820 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 851744 T2820 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 851746 T2820 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 851757 T2820 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 851760 T2820 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 851762 T2820 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 851763 T2820 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 851763 T2820 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 851764 T2820 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 851764 T2820 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 851764 T2820 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 851765 T2820 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 851765 T2820 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371894643583/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3/
[junit4:junit4] 2> 851765 T2820 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@75baec8e
[junit4:junit4] 2> 851765 T2820 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 851766 T2820 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3
[junit4:junit4] 2> 851766 T2820 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3/index/
[junit4:junit4] 2> 851766 T2820 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 851766 T2820 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3/index
[junit4:junit4] 2> 851769 T2820 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6a7166f9; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 851769 T2820 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 851771 T2820 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 851771 T2820 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 851772 T2820 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 851772 T2820 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 851773 T2820 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 851773 T2820 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 851773 T2820 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 851773 T2820 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 851773 T2820 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 851776 T2820 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 851778 T2820 oass.SolrIndexSearcher.<init> Opening Searcher@53edfc92 main
[junit4:junit4] 2> 851779 T2820 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 851779 T2820 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 851783 T2821 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@53edfc92 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 851784 T2820 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 851784 T2820 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42901/la_h/dt collection:collection1 shard:shard1
[junit4:junit4] 2> 851787 T2820 oasc.ZkController.register We are http://127.0.0.1:42901/la_h/dt/collection1/ and leader is http://127.0.0.1:46423/la_h/dt/collection1/
[junit4:junit4] 2> 851787 T2820 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42901/la_h/dt
[junit4:junit4] 2> 851787 T2820 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 851787 T2820 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C3480 name=collection1 org.apache.solr.core.SolrCore@6cd30d3e url=http://127.0.0.1:42901/la_h/dt/collection1 node=127.0.0.1:42901_la_h%2Fdt C3480_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:42901_la_h%2Fdt, base_url=http://127.0.0.1:42901/la_h/dt}
[junit4:junit4] 2> 851787 T2822 C3480 P42901 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 851788 T2822 C3480 P42901 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 851788 T2820 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 851788 T2822 C3480 P42901 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 851789 T2822 C3480 P42901 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 851789 T2744 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 851790 T2744 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 851790 T2744 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 851791 T2822 C3480 P42901 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 851794 T2780 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 851849 T2744 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 851851 T2744 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40283
[junit4:junit4] 2> 851851 T2744 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 851851 T2744 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 851852 T2744 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371894646176
[junit4:junit4] 2> 851852 T2744 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371894646176/solr.xml
[junit4:junit4] 2> 851852 T2744 oasc.CoreContainer.<init> New CoreContainer 1802675918
[junit4:junit4] 2> 851852 T2744 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371894646176/'
[junit4:junit4] 2> 851853 T2744 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371894646176/'
[junit4:junit4] 2> 851903 T2744 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 851903 T2744 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 851903 T2744 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 851904 T2744 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 851904 T2744 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 851904 T2744 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 851904 T2744 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 851904 T2744 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 851905 T2744 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 851905 T2744 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 851908 T2744 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 851909 T2744 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55758/solr
[junit4:junit4] 2> 851909 T2744 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 851910 T2744 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 851912 T2834 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@107aca94 name:ZooKeeperConnection Watcher:127.0.0.1:55758 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 851912 T2744 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 851914 T2744 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 851916 T2744 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 851918 T2836 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@743a122c name:ZooKeeperConnection Watcher:127.0.0.1:55758/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 851918 T2744 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 851921 T2744 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C3481 name=collection1 org.apache.solr.core.SolrCore@bebc2b url=http://127.0.0.1:52290/la_h/dt/collection1 node=127.0.0.1:52290_la_h%2Fdt C3481_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:52290_la_h%2Fdt, base_url=http://127.0.0.1:52290/la_h/dt}
[junit4:junit4] 2> 852203 T2805 C3481 P52290 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:46423/la_h/dt/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 852203 T2805 C3481 P52290 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52290/la_h/dt START replicas=[http://127.0.0.1:46423/la_h/dt/collection1/] nUpdates=100
[junit4:junit4] 2> 852203 T2805 C3481 P52290 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 852204 T2805 C3481 P52290 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 852204 T2805 C3481 P52290 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 852204 T2805 C3481 P52290 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 852204 T2805 C3481 P52290 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 852204 T2805 C3481 P52290 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:46423/la_h/dt/collection1/. core=collection1
[junit4:junit4] 2> 852205 T2805 C3481 P52290 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C3482 name=collection1 org.apache.solr.core.SolrCore@61d11f90 url=http://127.0.0.1:46423/la_h/dt/collection1 node=127.0.0.1:46423_la_h%2Fdt C3482_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:46423_la_h%2Fdt, base_url=http://127.0.0.1:46423/la_h/dt, leader=true}
[junit4:junit4] 2> 852208 T2779 C3482 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 852211 T2782 C3482 P46423 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 852213 T2782 C3482 P46423 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 852214 T2782 C3482 P46423 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 852223 T2782 C3482 P46423 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 852224 T2782 C3482 P46423 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 852224 T2782 C3482 P46423 oass.SolrIndexSearcher.<init> Opening Searcher@6e19cdf7 realtime
[junit4:junit4] 2> 852224 T2782 C3482 P46423 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 852225 T2782 C3482 P46423 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 14
[junit4:junit4] 2> 852225 T2805 C3481 P52290 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 852225 T2805 C3481 P52290 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 852227 T2779 C3482 P46423 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 852227 T2779 C3482 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 852228 T2805 C3481 P52290 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 852228 T2805 C3481 P52290 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 852228 T2805 C3481 P52290 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 852230 T2782 C3482 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 852230 T2805 C3481 P52290 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 852231 T2805 C3481 P52290 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2/index.20130622025046614
[junit4:junit4] 2> 852231 T2805 C3481 P52290 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2/index.20130622025046614 lockFactory=org.apache.lucene.store.NativeFSLockFactory@56104331; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 852234 T2781 C3482 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 852235 T2805 C3481 P52290 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 852244 T2805 C3481 P52290 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 852244 T2805 C3481 P52290 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 852246 T2805 C3481 P52290 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17f4673c; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17f4673c; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 852246 T2805 C3481 P52290 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 852246 T2805 C3481 P52290 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 852247 T2805 C3481 P52290 oass.SolrIndexSearcher.<init> Opening Searcher@7a91049e main
[junit4:junit4] 2> 852247 T2804 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7a91049e main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 852248 T2805 C3481 P52290 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2/index.20130622025046614 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2/index.20130622025046614;done=true>>]
[junit4:junit4] 2> 852248 T2805 C3481 P52290 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2/index.20130622025046614
[junit4:junit4] 2> 852248 T2805 C3481 P52290 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2/index.20130622025046614
[junit4:junit4] 2> 852248 T2805 C3481 P52290 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 852249 T2805 C3481 P52290 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 852249 T2805 C3481 P52290 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 852249 T2805 C3481 P52290 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 852251 T2805 C3481 P52290 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 852615 T2767 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 852616 T2767 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42901_la_h%2Fdt",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42901/la_h/dt"}
[junit4:junit4] 2> 852618 T2767 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52290_la_h%2Fdt",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52290/la_h/dt"}
[junit4:junit4] 2> 852634 T2773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 852634 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 852634 T2836 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 852634 T2786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 852634 T2766 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 852634 T2802 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 852795 T2780 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 852795 T2780 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=4&state=recovering&nodeName=127.0.0.1:42901_la_h%252Fdt&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 852923 T2744 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40283_la_h%2Fdt
[junit4:junit4] 2> 852924 T2744 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40283_la_h%2Fdt
[junit4:junit4] 2> 852926 T2773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 852926 T2802 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 852926 T2786 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 852927 T2766 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 852927 T2766 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 852927 T2819 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 852927 T2802 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 852928 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 852927 T2786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 852927 T2836 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 852928 T2773 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 852929 T2836 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 852933 T2839 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 852933 T2839 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 854140 T2767 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 854141 T2767 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40283_la_h%2Fdt",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40283/la_h/dt"}
[junit4:junit4] 2> 854141 T2767 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 854141 T2767 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 854145 T2773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 854145 T2836 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 854145 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 854145 T2766 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 854145 T2786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 854145 T2802 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> ASYNC NEW_CORE C3483 name=collection1 org.apache.solr.core.SolrCore@6cd30d3e url=http://127.0.0.1:42901/la_h/dt/collection1 node=127.0.0.1:42901_la_h%2Fdt C3483_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:42901_la_h%2Fdt, base_url=http://127.0.0.1:42901/la_h/dt}
[junit4:junit4] 2> 854796 T2822 C3483 P42901 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:46423/la_h/dt/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 854796 T2822 C3483 P42901 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:42901/la_h/dt START replicas=[http://127.0.0.1:46423/la_h/dt/collection1/] nUpdates=100
[junit4:junit4] 2> 854797 T2822 C3483 P42901 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 854797 T2822 C3483 P42901 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 854797 T2822 C3483 P42901 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 854797 T2822 C3483 P42901 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 854798 T2822 C3483 P42901 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 854798 T2822 C3483 P42901 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:46423/la_h/dt/collection1/. core=collection1
[junit4:junit4] 2> 854798 T2822 C3483 P42901 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 854799 T2782 C3482 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 854801 T2779 C3482 P46423 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 854813 T2779 C3482 P46423 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 854814 T2779 C3482 P46423 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 854815 T2779 C3482 P46423 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 854815 T2779 C3482 P46423 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 14
[junit4:junit4] 2> 854815 T2822 C3483 P42901 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 854816 T2822 C3483 P42901 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 854817 T2782 C3482 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 854818 T2822 C3483 P42901 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4] 2> 854818 T2822 C3483 P42901 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 854818 T2822 C3483 P42901 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 854820 T2781 C3482 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=3&version=2} status=0 QTime=0
[junit4:junit4] 2> 854820 T2822 C3483 P42901 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 854821 T2822 C3483 P42901 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3/index.20130622025049204
[junit4:junit4] 2> 854821 T2822 C3483 P42901 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3/index.20130622025049204 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3d91d35f; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 854824 T2779 C3482 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/replication params={file=segments_3&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=3} status=0 QTime=0
[junit4:junit4] 2> 854825 T2822 C3483 P42901 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 854827 T2822 C3483 P42901 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 854827 T2822 C3483 P42901 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 854829 T2822 C3483 P42901 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6a7166f9; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6a7166f9; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 854829 T2822 C3483 P42901 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 854829 T2822 C3483 P42901 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 854829 T2822 C3483 P42901 oass.SolrIndexSearcher.<init> Opening Searcher@19f8b674 main
[junit4:junit4] 2> 854830 T2821 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19f8b674 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 854831 T2822 C3483 P42901 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3/index.20130622025049204 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3/index.20130622025049204;done=true>>]
[junit4:junit4] 2> 854831 T2822 C3483 P42901 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3/index.20130622025049204
[junit4:junit4] 2> 854831 T2822 C3483 P42901 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3/index.20130622025049204
[junit4:junit4] 2> 854831 T2822 C3483 P42901 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 854831 T2822 C3483 P42901 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 854831 T2822 C3483 P42901 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 854832 T2822 C3483 P42901 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 854854 T2822 C3483 P42901 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 854934 T2839 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371894646176/collection1
[junit4:junit4] 2> 854934 T2839 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 854935 T2839 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 854935 T2839 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 854936 T2839 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371894646176/collection1/'
[junit4:junit4] 2> 854937 T2839 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1371894646176/collection1/lib/README' to classloader
[junit4:junit4] 2> 854937 T2839 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1371894646176/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 854973 T2839 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 855007 T2839 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 855009 T2839 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 855012 T2839 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 855330 T2839 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 855335 T2839 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 855337 T2839 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 855350 T2839 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 855353 T2839 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 855356 T2839 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 855357 T2839 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 855357 T2839 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 855357 T2839 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 855358 T2839 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 855359 T2839 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 855359 T2839 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 855359 T2839 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371894646176/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4/
[junit4:junit4] 2> 855359 T2839 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@75baec8e
[junit4:junit4] 2> 855360 T2839 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
[junit4:junit4] 2> 855360 T2839 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4
[junit4:junit4] 2> 855360 T2839 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4/index/
[junit4:junit4] 2> 855360 T2839 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 855361 T2839 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4/index
[junit4:junit4] 2> 855364 T2839 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@420e92a2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 855364 T2839 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 855365 T2839 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 855366 T2839 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 855366 T2839 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 855366 T2839 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 855367 T2839 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 855367 T2839 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 855368 T2839 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 855369 T2839 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 855369 T2839 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 855371 T2839 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 855373 T2839 oass.SolrIndexSearcher.<init> Opening Searcher@3734c064 main
[junit4:junit4] 2> 855374 T2839 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 855374 T2839 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 855378 T2842 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3734c064 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 855380 T2839 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 855380 T2839 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40283/la_h/dt collection:collection1 shard:shard1
[junit4:junit4] 2> 855383 T2839 oasc.ZkController.register We are http://127.0.0.1:40283/la_h/dt/collection1/ and leader is http://127.0.0.1:46423/la_h/dt/collection1/
[junit4:junit4] 2> 855383 T2839 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40283/la_h/dt
[junit4:junit4] 2> 855384 T2839 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 855384 T2839 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C3484 name=collection1 org.apache.solr.core.SolrCore@22b940eb url=http://127.0.0.1:40283/la_h/dt/collection1 node=127.0.0.1:40283_la_h%2Fdt C3484_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:40283_la_h%2Fdt, base_url=http://127.0.0.1:40283/la_h/dt}
[junit4:junit4] 2> 855384 T2843 C3484 P40283 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 855385 T2843 C3484 P40283 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 855385 T2839 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 855385 T2843 C3484 P40283 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 855385 T2843 C3484 P40283 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 855386 T2744 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 855386 T2744 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 855386 T2744 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 855387 T2843 C3484 P40283 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 855390 T2744 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 855391 T2781 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 855391 T2744 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 855391 T2744 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 855392 T2744 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 855649 T2767 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 855650 T2767 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42901_la_h%2Fdt",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42901/la_h/dt"}
[junit4:junit4] 2> 855653 T2767 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40283_la_h%2Fdt",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40283/la_h/dt"}
[junit4:junit4] 2> 855656 T2773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 855656 T2802 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 855656 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 855656 T2836 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 855656 T2786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 855656 T2766 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 856391 T2781 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 856392 T2781 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:40283_la_h%252Fdt&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 856393 T2744 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 857395 T2744 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> C3484_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:40283_la_h%2Fdt, base_url=http://127.0.0.1:40283/la_h/dt}
[junit4:junit4] 2> 858393 T2843 C3484 P40283 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:46423/la_h/dt/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 858393 T2843 C3484 P40283 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:40283/la_h/dt START replicas=[http://127.0.0.1:46423/la_h/dt/collection1/] nUpdates=100
[junit4:junit4] 2> 858393 T2843 C3484 P40283 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 858393 T2843 C3484 P40283 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 858393 T2843 C3484 P40283 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 858394 T2843 C3484 P40283 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 858394 T2843 C3484 P40283 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 858394 T2843 C3484 P40283 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:46423/la_h/dt/collection1/. core=collection1
[junit4:junit4] 2> 858394 T2843 C3484 P40283 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C3485 name=collection1 org.apache.solr.core.SolrCore@61d11f90 url=http://127.0.0.1:46423/la_h/dt/collection1 node=127.0.0.1:46423_la_h%2Fdt C3485_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:46423_la_h%2Fdt, base_url=http://127.0.0.1:46423/la_h/dt, leader=true}
[junit4:junit4] 2> 858395 T2781 C3485 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 858396 T2744 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 858397 T2779 C3485 P46423 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 858403 T2779 C3485 P46423 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 858404 T2779 C3485 P46423 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 858404 T2779 C3485 P46423 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 858404 T2779 C3485 P46423 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4] 2> 858405 T2843 C3484 P40283 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 858405 T2843 C3484 P40283 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 858407 T2782 C3485 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 858408 T2843 C3484 P40283 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4] 2> 858408 T2843 C3484 P40283 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 858408 T2843 C3484 P40283 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 858410 T2781 C3485 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=4&version=2} status=0 QTime=1
[junit4:junit4] 2> 858410 T2843 C3484 P40283 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 858411 T2843 C3484 P40283 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4/index.20130622025052793
[junit4:junit4] 2> 858411 T2843 C3484 P40283 oash.SnapPuller.fetchLatestIndex Starting download to NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4/index.20130622025052793 lockFactory=org.apache.lucene.store.NativeFSLockFactory@632d7b0c; maxCacheMB=48.0 maxMergeSizeMB=4.0) fullCopy=false
[junit4:junit4] 2> 858414 T2779 C3485 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/replication params={file=segments_4&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=4} status=0 QTime=1
[junit4:junit4] 2> 858415 T2843 C3484 P40283 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 858417 T2843 C3484 P40283 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 858417 T2843 C3484 P40283 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 858419 T2843 C3484 P40283 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@420e92a2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@420e92a2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 858420 T2843 C3484 P40283 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 858420 T2843 C3484 P40283 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 858420 T2843 C3484 P40283 oass.SolrIndexSearcher.<init> Opening Searcher@4efd909c main
[junit4:junit4] 2> 858421 T2842 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4efd909c main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 858421 T2843 C3484 P40283 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4/index.20130622025052793 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4/index.20130622025052793;done=true>>]
[junit4:junit4] 2> 858421 T2843 C3484 P40283 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4/index.20130622025052793
[junit4:junit4] 2> 858421 T2843 C3484 P40283 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4/index.20130622025052793
[junit4:junit4] 2> 858422 T2843 C3484 P40283 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 858422 T2843 C3484 P40283 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 858422 T2843 C3484 P40283 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 858422 T2843 C3484 P40283 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 858444 T2843 C3484 P40283 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 858663 T2767 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 858664 T2767 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40283_la_h%2Fdt",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40283/la_h/dt"}
[junit4:junit4] 2> 858669 T2766 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 858669 T2786 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 858669 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 858669 T2836 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 858669 T2802 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 858669 T2773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 859398 T2744 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 859399 T2744 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C3486 name=collection1 org.apache.solr.core.SolrCore@7103a27a url=http://127.0.0.1:48059/la_h/dt/collection1 node=127.0.0.1:48059_la_h%2Fdt C3486_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:48059_la_h%2Fdt, base_url=http://127.0.0.1:48059/la_h/dt, leader=true}
[junit4:junit4] 2> 859405 T2757 C3486 P48059 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 859408 T2757 C3486 P48059 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@415b13b9; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> 859409 T2757 C3486 P48059 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 859413 T2757 C3486 P48059 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@415b13b9; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@415b13b9; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> 859414 T2757 C3486 P48059 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 859415 T2757 C3486 P48059 oass.SolrIndexSearcher.<init> Opening Searcher@14acfd3c main
[junit4:junit4] 2> 859415 T2757 C3486 P48059 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 859415 T2770 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@14acfd3c main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 859416 T2757 C3486 P48059 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4] 2> ASYNC NEW_CORE C3487 name=collection1 org.apache.solr.core.SolrCore@bebc2b url=http://127.0.0.1:52290/la_h/dt/collection1 node=127.0.0.1:52290_la_h%2Fdt C3487_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52290_la_h%2Fdt, base_url=http://127.0.0.1:52290/la_h/dt}
[junit4:junit4] 2> 859430 T2798 C3487 P52290 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:46423/la_h/dt/collection1/, StdNode: http://127.0.0.1:52290/la_h/dt/collection1/, StdNode: http://127.0.0.1:42901/la_h/dt/collection1/, StdNode: http://127.0.0.1:40283/la_h/dt/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 859433 T2797 C3487 P52290 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C3488 name=collection1 org.apache.solr.core.SolrCore@61d11f90 url=http://127.0.0.1:46423/la_h/dt/collection1 node=127.0.0.1:46423_la_h%2Fdt C3488_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:46423_la_h%2Fdt, base_url=http://127.0.0.1:46423/la_h/dt, leader=true}
[junit4:junit4] 2> 859433 T2779 C3488 P46423 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C3489 name=collection1 org.apache.solr.core.SolrCore@6cd30d3e url=http://127.0.0.1:42901/la_h/dt/collection1 node=127.0.0.1:42901_la_h%2Fdt C3489_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:42901_la_h%2Fdt, base_url=http://127.0.0.1:42901/la_h/dt}
[junit4:junit4] 2> 859438 T2814 C3489 P42901 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C3490 name=collection1 org.apache.solr.core.SolrCore@22b940eb url=http://127.0.0.1:40283/la_h/dt/collection1 node=127.0.0.1:40283_la_h%2Fdt C3490_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:40283_la_h%2Fdt, base_url=http://127.0.0.1:40283/la_h/dt}
[junit4:junit4] 2> 859438 T2830 C3490 P40283 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 859453 T2797 C3487 P52290 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17f4673c; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17f4673c; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 859454 T2797 C3487 P52290 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 859455 T2797 C3487 P52290 oass.SolrIndexSearcher.<init> Opening Searcher@59ec09e6 main
[junit4:junit4] 2> 859455 T2797 C3487 P52290 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 859456 T2804 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@59ec09e6 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 859456 T2797 C3487 P52290 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 23
[junit4:junit4] 2> 859483 T2830 C3490 P40283 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@420e92a2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@420e92a2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 859483 T2814 C3489 P42901 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6a7166f9; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6a7166f9; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 859483 T2779 C3488 P46423 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=4
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 859485 T2779 C3488 P46423 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 859485 T2814 C3489 P42901 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 859484 T2830 C3490 P40283 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 859485 T2779 C3488 P46423 oass.SolrIndexSearcher.<init> Opening Searcher@6716fc46 main
[junit4:junit4] 2> 859486 T2814 C3489 P42901 oass.SolrIndexSearcher.<init> Opening Searcher@665ee618 main
[junit4:junit4] 2> 859486 T2830 C3490 P40283 oass.SolrIndexSearcher.<init> Opening Searcher@44ff3f27 main
[junit4:junit4] 2> 859486 T2779 C3488 P46423 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 859487 T2830 C3490 P40283 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 859487 T2821 oasc.JmxMonitoredMap.put WARN Failed to register info bean: searcher javax.management.InstanceAlreadyExistsException: solr/collection1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4] 2> at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:299)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1846)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1723)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2>
[junit4:junit4] 2> 859487 T2814 C3489 P42901 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 859487 T2842 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@44ff3f27 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 859487 T2788 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6716fc46 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 859488 T2830 C3490 P40283 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 50
[junit4:junit4] 2> 859488 T2821 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@665ee618 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 859489 T2779 C3488 P46423 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 56
[junit4:junit4] 2> 859489 T2814 C3489 P42901 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 51
[junit4:junit4] 2> 859489 T2798 C3487 P52290 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 59
[junit4:junit4] 2> 859490 T2744 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 859492 T2782 C3488 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 859494 T2795 C3487 P52290 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4] 2> 859495 T2815 C3489 P42901 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 859496 T2832 C3490 P40283 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> ASYNC NEW_CORE C3491 name=collection1 org.apache.solr.core.SolrCore@7103a27a url=http://127.0.0.1:48059/la_h/dt/collection1 node=127.0.0.1:48059_la_h%2Fdt C3491_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:48059_la_h%2Fdt, base_url=http://127.0.0.1:48059/la_h/dt, leader=true}
[junit4:junit4] 2> 861499 T2757 C3491 P48059 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1438535810686124032)} 0 1
[junit4:junit4] 2> 861508 T2797 C3487 P52290 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={update.distrib=FROMLEADER&_version_=-1438535810691366912&update.from=http://127.0.0.1:46423/la_h/dt/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438535810691366912)} 0 1
[junit4:junit4] 2> 861508 T2831 C3490 P40283 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={update.distrib=FROMLEADER&_version_=-1438535810691366912&update.from=http://127.0.0.1:46423/la_h/dt/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438535810691366912)} 0 1
[junit4:junit4] 2> 861512 T2814 C3489 P42901 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={update.distrib=FROMLEADER&_version_=-1438535810691366912&update.from=http://127.0.0.1:46423/la_h/dt/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438535810691366912)} 0 2
[junit4:junit4] 2> 861513 T2779 C3488 P46423 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1438535810691366912)} 0 10
[junit4:junit4] 2> 861513 T2798 C3487 P52290 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 12
[junit4:junit4] 2> 861518 T2758 C3491 P48059 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={wt=javabin&version=2} {add=[0 (1438535810703949824)]} 0 2
[junit4:junit4] 2> 861526 T2795 C3487 P52290 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={distrib.from=http://127.0.0.1:46423/la_h/dt/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438535810708144128)]} 0 2
[junit4:junit4] 2> 861526 T2832 C3490 P40283 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={distrib.from=http://127.0.0.1:46423/la_h/dt/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438535810708144128)]} 0 2
[junit4:junit4] 2> 861526 T2815 C3489 P42901 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={distrib.from=http://127.0.0.1:46423/la_h/dt/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438535810708144128)]} 0 2
[junit4:junit4] 2> 861527 T2781 C3488 P46423 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={wt=javabin&version=2} {add=[0 (1438535810708144128)]} 0 8
[junit4:junit4] 2> 861530 T2759 C3491 P48059 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={wt=javabin&version=2} {add=[1 (1438535810717581312)]} 0 1
[junit4:junit4] 2> 861535 T2829 C3490 P40283 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={distrib.from=http://127.0.0.1:46423/la_h/dt/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438535810720727040)]} 0 1
[junit4:junit4] 2> 861535 T2796 C3487 P52290 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={distrib.from=http://127.0.0.1:46423/la_h/dt/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438535810720727040)]} 0 1
[junit4:junit4] 2> 861535 T2812 C3489 P42901 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={distrib.from=http://127.0.0.1:46423/la_h/dt/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438535810720727040)]} 0 1
[junit4:junit4] 2> 861536 T2780 C3488 P46423 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={wt=javabin&version=2} {add=[1 (1438535810720727040)]} 0 4
[junit4:junit4] 2> 861539 T2756 C3491 P48059 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={wt=javabin&version=2} {add=[2 (1438535810727018496)]} 0 1
[junit4:junit4] 2> 861543 T2779 C3488 P46423 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:52290/la_h/dt/collection1/]
[junit4:junit4] 2> 861543 T2779 C3488 P46423 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:52290/la_h/dt/collection1/ against:[http://127.0.0.1:52290/la_h/dt/collection1/] result:true
[junit4:junit4] 2> 861543 T2779 C3488 P46423 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:42901/la_h/dt/collection1/ against:[http://127.0.0.1:52290/la_h/dt/collection1/] result:false
[junit4:junit4] 2> 861543 T2779 C3488 P46423 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:40283/la_h/dt/collection1/ against:[http://127.0.0.1:52290/la_h/dt/collection1/] result:false
[junit4:junit4] 2> 861547 T2830 C3490 P40283 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={distrib.from=http://127.0.0.1:46423/la_h/dt/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:52290/la_h/dt/collection1/&wt=javabin&version=2} {add=[2 (1438535810733309952)]} 0 1
[junit4:junit4] 2> 861547 T2814 C3489 P42901 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={distrib.from=http://127.0.0.1:46423/la_h/dt/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:52290/la_h/dt/collection1/&wt=javabin&version=2} {add=[2 (1438535810733309952)]} 0 1
[junit4:junit4] 2> 861548 T2779 C3488 P46423 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={distrib.from=http://127.0.0.1:42901/la_h/dt/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:52290/la_h/dt/collection1/&wt=javabin&version=2} {add=[2 (1438535810733309952)]} 0 6
[junit4:junit4] 2> 861549 T2813 C3489 P42901 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={test.distrib.skip.servers=http://127.0.0.1:52290/la_h/dt/collection1/&wt=javabin&version=2} {add=[2]} 0 9
[junit4:junit4] 2> 861552 T2757 C3491 P48059 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={wt=javabin&version=2} {add=[3 (1438535810740649984)]} 0 1
[junit4:junit4] 2> 861556 T2781 C3488 P46423 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:52290/la_h/dt/collection1/]
[junit4:junit4] 2> 861556 T2781 C3488 P46423 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:52290/la_h/dt/collection1/ against:[http://127.0.0.1:52290/la_h/dt/collection1/] result:true
[junit4:junit4] 2> 861556 T2781 C3488 P46423 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:42901/la_h/dt/collection1/ against:[http://127.0.0.1:52290/la_h/dt/collection1/] result:false
[junit4:junit4] 2> 861557 T2781 C3488 P46423 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:40283/la_h/dt/collection1/ against:[http://127.0.0.1:52290/la_h/dt/collection1/] result:false
[junit4:junit4] 2> 861561 T2829 C3490 P40283 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={distrib.from=http://127.0.0.1:46423/la_h/dt/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:52290/la_h/dt/collection1/&wt=javabin&version=2} {add=[3 (1438535810746941440)]} 0 1
[junit4:junit4] 2> 861561 T2815 C3489 P42901 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={distrib.from=http://127.0.0.1:46423/la_h/dt/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:52290/la_h/dt/collection1/&wt=javabin&version=2} {add=[3 (1438535810746941440)]} 0 1
[junit4:junit4] 2> 861561 T2781 C3488 P46423 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={distrib.from=http://127.0.0.1:40283/la_h/dt/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:52290/la_h/dt/collection1/&wt=javabin&version=2} {add=[3 (1438535810746941440)]} 0 6
[junit4:junit4] 2> 861562 T2832 C3490 P40283 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={test.distrib.skip.servers=http://127.0.0.1:52290/la_h/dt/collection1/&test.distrib.skip.servers=http://127.0.0.1:42901/la_h/dt/collection1/&wt=javabin&version=2} {add=[3]} 0 9
[junit4:junit4] 2> 861565 T2758 C3491 P48059 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 861694 T2758 C3491 P48059 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@415b13b9; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@415b13b9; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> 861695 T2758 C3491 P48059 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 861701 T2758 C3491 P48059 oass.SolrIndexSearcher.<init> Opening Searcher@21f53acd main
[junit4:junit4] 2> 861701 T2758 C3491 P48059 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 861703 T2770 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@21f53acd main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 861703 T2758 C3491 P48059 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 139
[junit4:junit4] 2> 861706 T2780 C3488 P46423 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:46423/la_h/dt/collection1/, StdNode: http://127.0.0.1:52290/la_h/dt/collection1/, StdNode: http://127.0.0.1:42901/la_h/dt/collection1/, StdNode: http://127.0.0.1:40283/la_h/dt/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> 861709 T2782 C3488 P46423 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 861709 T2830 C3490 P40283 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 861709 T2812 C3489 P42901 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 861709 T2797 C3487 P52290 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 861764 T2797 C3487 P52290 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17f4673c; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@17f4673c; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 861764 T2797 C3487 P52290 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 861769 T2797 C3487 P52290 oass.SolrIndexSearcher.<init> Opening Searcher@637049f1 main
[junit4:junit4] 2> 861770 T2797 C3487 P52290 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 861771 T2804 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@637049f1 main{StandardDirectoryReader(segments_4:3:nrt _0(5.0):C2)}
[junit4:junit4] 2> 861772 T2797 C3487 P52290 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 63
[junit4:junit4] 2> 861773 T2830 C3490 P40283 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@420e92a2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty4/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@420e92a2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> 861774 T2830 C3490 P40283 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 861773 T2782 C3488 P46423 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@16a90b05; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_6,generation=6}
[junit4:junit4] 2> 861775 T2782 C3488 P46423 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 861773 T2812 C3489 P42901 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6a7166f9; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6a7166f9; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_5,generation=5}
[junit4:junit4] 2> 861777 T2812 C3489 P42901 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 861778 T2830 C3490 P40283 oass.SolrIndexSearcher.<init> Opening Searcher@33d45e34 main
[junit4:junit4] 2> 861779 T2830 C3490 P40283 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 861780 T2842 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@33d45e34 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 861781 T2830 C3490 P40283 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 72
[junit4:junit4] 2> 861782 T2812 C3489 P42901 oass.SolrIndexSearcher.<init> Opening Searcher@1e1ef02d main
[junit4:junit4] 2> 861782 T2782 C3488 P46423 oass.SolrIndexSearcher.<init> Opening Searcher@78853f2e main
[junit4:junit4] 2> 861783 T2812 C3489 P42901 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 861784 T2821 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e1ef02d main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 861783 T2782 C3488 P46423 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 861784 T2812 C3489 P42901 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 75
[junit4:junit4] 2> 861784 T2788 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@78853f2e main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 861786 T2782 C3488 P46423 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 78
[junit4:junit4] 2> 861787 T2780 C3488 P46423 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 81
[junit4:junit4] 2> 861787 T2744 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 861789 T2744 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 861790 T2744 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 861793 T2779 C3488 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 861797 T2798 C3487 P52290 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=1
[junit4:junit4] 2> 861799 T2814 C3489 P42901 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 861802 T2831 C3490 P40283 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 861803 T2744 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 861810 T2813 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4] 2> 861811 T2813 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 861819 T2781 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4] 2> 861819 T2781 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46423/la_h/dt/collection1/
[junit4:junit4] 2> 861820 T2781 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:46423/la_h/dt START replicas=[http://127.0.0.1:52290/la_h/dt/collection1/, http://127.0.0.1:42901/la_h/dt/collection1/, http://127.0.0.1:40283/la_h/dt/collection1/] nUpdates=100
[junit4:junit4] 2> 861822 T2795 C3487 P52290 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 861823 T2829 C3490 P40283 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 861823 T2781 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:46423/la_h/dt Received 3 versions from 127.0.0.1:52290/la_h/dt/collection1/
[junit4:junit4] 2> 861822 T2815 C3489 P42901 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 861824 T2781 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:46423/la_h/dt Our versions are newer. ourLowThreshold=1438535810691366912 otherHigh=1438535810720727040
[junit4:junit4] 2> 861825 T2781 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:46423/la_h/dt Received 5 versions from 127.0.0.1:40283/la_h/dt/collection1/
[junit4:junit4] 2> 861825 T2781 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:46423/la_h/dt Our versions are newer. ourLowThreshold=1438535810691366912 otherHigh=1438535810733309952
[junit4:junit4] 2> 861825 T2781 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:46423/la_h/dt Received 5 versions from 127.0.0.1:42901/la_h/dt/collection1/
[junit4:junit4] 2> 861826 T2781 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:46423/la_h/dt Our versions are newer. ourLowThreshold=1438535810691366912 otherHigh=1438535810733309952
[junit4:junit4] 2> 861826 T2781 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:46423/la_h/dt DONE. sync succeeded
[junit4:junit4] 2> 861827 T2781 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 861827 T2781 oasc.SyncStrategy.syncToMe http://127.0.0.1:46423/la_h/dt/collection1/: try and ask http://127.0.0.1:52290/la_h/dt/collection1/ to sync
[junit4:junit4] 2> 861828 T2781 oasc.SyncStrategy.syncToMe http://127.0.0.1:46423/la_h/dt/collection1/: try and ask http://127.0.0.1:42901/la_h/dt/collection1/ to sync
[junit4:junit4] 2> 861829 T2781 oasc.SyncStrategy.syncToMe http://127.0.0.1:46423/la_h/dt/collection1/: try and ask http://127.0.0.1:40283/la_h/dt/collection1/ to sync
[junit4:junit4] 2> 861830 T2796 C3487 P52290 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52290/la_h/dt START replicas=[http://127.0.0.1:46423/la_h/dt/collection1/] nUpdates=100
[junit4:junit4] 2> 861830 T2812 C3489 P42901 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:42901/la_h/dt START replicas=[http://127.0.0.1:46423/la_h/dt/collection1/] nUpdates=100
[junit4:junit4] 2> 861831 T2832 C3490 P40283 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:40283/la_h/dt START replicas=[http://127.0.0.1:46423/la_h/dt/collection1/] nUpdates=100
[junit4:junit4] 2> 861833 T2782 C3488 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 861833 T2780 C3488 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 861834 T2812 C3489 P42901 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:42901/la_h/dt Received 5 versions from 127.0.0.1:46423/la_h/dt/collection1/
[junit4:junit4] 2> 861834 T2779 C3488 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 861834 T2812 C3489 P42901 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:42901/la_h/dt Our versions are newer. ourLowThreshold=1438535810691366912 otherHigh=1438535810733309952
[junit4:junit4] 2> 861835 T2812 C3489 P42901 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:42901/la_h/dt DONE. sync succeeded
[junit4:junit4] 2> 861836 T2812 C3489 P42901 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/get params={sync=http://127.0.0.1:46423/la_h/dt/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=6
[junit4:junit4] 2> 861834 T2796 C3487 P52290 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:52290/la_h/dt Received 5 versions from 127.0.0.1:46423/la_h/dt/collection1/
[junit4:junit4] 2> 861836 T2781 oasc.SyncStrategy.syncToMe http://127.0.0.1:46423/la_h/dt/collection1/: sync completed with http://127.0.0.1:42901/la_h/dt/collection1/
[junit4:junit4] 2> 861835 T2832 C3490 P40283 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:40283/la_h/dt Received 5 versions from 127.0.0.1:46423/la_h/dt/collection1/
[junit4:junit4] 2> 861837 T2796 C3487 P52290 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:52290/la_h/dt Requesting updates from 127.0.0.1:46423/la_h/dt/collection1/n=2 versions=[1438535810746941440, 1438535810733309952]
[junit4:junit4] 2> 861838 T2832 C3490 P40283 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:40283/la_h/dt Our versions are newer. ourLowThreshold=1438535810691366912 otherHigh=1438535810733309952
[junit4:junit4] 2> 861839 T2832 C3490 P40283 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:40283/la_h/dt DONE. sync succeeded
[junit4:junit4] 2> 861839 T2832 C3490 P40283 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/get params={sync=http://127.0.0.1:46423/la_h/dt/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=8
[junit4:junit4] 2> 861840 T2781 oasc.SyncStrategy.syncToMe http://127.0.0.1:46423/la_h/dt/collection1/: sync completed with http://127.0.0.1:40283/la_h/dt/collection1/
[junit4:junit4] 2> 861840 T2782 C3488 P46423 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/get params={distrib=false&getUpdates=1438535810746941440,1438535810733309952&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 861846 T2796 C3487 P52290 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1438535810733309952), 3 (1438535810746941440)]} 0 5
[junit4:junit4] 2> 861847 T2796 C3487 P52290 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52290/la_h/dt DONE. sync succeeded
[junit4:junit4] 2> 861847 T2796 C3487 P52290 oasc.SolrCore.execute [collection1] webapp=/la_h/dt path=/get params={sync=http://127.0.0.1:46423/la_h/dt/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=17
[junit4:junit4] 2> 861848 T2781 oasc.SyncStrategy.syncToMe http://127.0.0.1:46423/la_h/dt/collection1/: sync completed with http://127.0.0.1:52290/la_h/dt/collection1/
[junit4:junit4] 2> 861848 T2781 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=29
[junit4:junit4] 2> 861849 T2813 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=39
[junit4:junit4] 2> 861850 T2744 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 861851 T2744 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 861852 T2744 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 861853 T2744 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 861856 T2756 C3491 P48059 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 861894 T2756 C3491 P48059 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@415b13b9; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371894634529/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@415b13b9; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_4,generation=4}
[junit4:junit4] 2> 861894 T2756 C3491 P48059 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 861895 T2756 C3491 P48059 oass.SolrIndexSearcher.<init> Opening Searcher@4095696 main
[junit4:junit4] 2> 861896 T2756 C3491 P48059 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 861913 T2770 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4095696 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> ASYNC NEW_CORE C3492 name=collection1 org.apache.solr.core.SolrCore@7103a27a url=http://127.0.0.1:48059/la_h/dt/collection1 node=127.0.0.1:48059_la_h%2Fdt C3492_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:48059_la_h%2Fdt, base_url=http://127.0.0.1:48059/la_h/dt, leader=true}
[junit4:junit4] 2> 861914 T2756 C3492 P48059 oasup.LogUpdateProcessor.finish [collection1] webapp=/la_h/dt path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 58
[junit4:junit4] 2> ASYNC NEW_CORE C3493 name=collection1 org.apache.solr.core.SolrCore@bebc2b url=http://127.0.0.1:52290/la_h/dt/collection1 node=127.0.0.1:52290_la_h%2Fdt C3493_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52290_la_h%2Fdt, base_url=http://127.0.0.1:52290/la_h/dt}
[junit4:junit4] 2> 861917 T2797 C3493 P52290 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:46423/la_h/dt/collection1/, StdNode: http://127.0.0.1:52290/la_h/dt/collection1/, StdNode: http://127.0.0.1:42901/la_h/dt/collection1/, StdNode: http://127
[...truncated too long message...]
nit4] 1> "shard":"shard1",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "node_name":"127.0.0.1:48059_la_h%2Fdt",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:48059/la_h/dt",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:42901_la_h%2Fdt (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:46423_la_h%2Fdt (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:40283_la_h%2Fdt (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:48059_la_h%2Fdt (0)
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/collections (2)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89908486768754700-5-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89908486768754702-2-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89908486768754698-4-n_0000000005 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (1)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:40283_la_h%2Fdt",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:40283/la_h/dt"}
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89908486768754691-1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:48059_la_h%2Fdt",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:48059/la_h/dt"}
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/election (4)
[junit4:junit4] 1> /solr/overseer_elect/election/89908486768754702-127.0.0.1:46423_la_h%2Fdt-n_0000000005 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89908486768754691-127.0.0.1:48059_la_h%2Fdt-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89908486768754700-127.0.0.1:40283_la_h%2Fdt-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89908486768754698-127.0.0.1:42901_la_h%2Fdt-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89908486768754691-127.0.0.1:48059_la_h%2Fdt-n_0000000000"}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=1D2543F64D8FF15C -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=fr_BE -Dtests.timezone=SystemV/MST7 -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 101s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:46423/la_h/dt/collection1lastClient and got 257 from http://127.0.0.1:42901/la_h/dt/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([1D2543F64D8FF15C:9CC3CDEE3AD09160]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 940681 T2744 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 100546 T2743 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
[junit4:junit4] 2> 940766 T2819 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 940767 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 940767 T2819 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 941842 T2836 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 941842 T2836 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 941842 T2836 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=107), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=107)), sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=fr_BE, timezone=SystemV/MST7
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=8,threads=1,free=55235312,total=164241408
[junit4:junit4] 2> NOTE: All tests run in this JVM: [DateFieldTest, SynonymTokenizerTest, SliceStateUpdateTest, DirectSolrConnectionTest, LukeRequestHandlerTest, TestLRUCache, TermsComponentTest, TestFuzzyAnalyzedSuggestions, SuggesterFSTTest, TestSchemaResource, DocValuesTest, BadCopyFieldTest, SchemaVersionSpecificBehaviorTest, FileBasedSpellCheckerTest, PeerSyncTest, WordBreakSolrSpellCheckerTest, ZkCLITest, BinaryUpdateRequestHandlerTest, TestConfig, UUIDFieldTest, TestPerFieldSimilarity, LegacyHTMLStripCharFilterTest, CSVRequestHandlerTest, TestFastWriter, AnalysisAfterCoreReloadTest, ResponseLogComponentTest, PreAnalyzedUpdateProcessorTest, JsonLoaderTest, RegexBoostProcessorTest, SystemInfoHandlerTest, PreAnalyzedFieldTest, TestBadConfig, TestSweetSpotSimilarityFactory, TestGroupingSearch, TestSolrXmlPersistence, SoftAutoCommitTest, TestIBSimilarityFactory, ReturnFieldsTest, PolyFieldTest, DocumentBuilderTest, ShardSplitTest, CoreAdminHandlerTest, TestJoin, TestCloudManagedSchema, LoggingHandlerTest, TestPhraseSuggestions, TestQueryUtils, TestStressRecovery, TestFieldCollectionResource, AutoCommitTest, TestDocSet, TestReloadAndDeleteDocs, LeaderElectionTest, TestSolrIndexConfig, RAMDirectoryFactoryTest, AliasIntegrationTest, PluginInfoTest, ChaosMonkeyNothingIsSafeTest, ClusterStateUpdateTest, SliceStateTest, SolrInfoMBeanTest, TestReplicationHandler, CircularListTest, UnloadDistributedZkTest, TestRemoteStreaming, ExternalFileFieldSortTest, ShardRoutingTest, QueryElevationComponentTest, TestReversedWildcardFilterFactory, SearchHandlerTest, BasicDistributedZkTest, SyncSliceTest]
[junit4:junit4] Completed on J0 in 101.85s, 1 test, 1 failure <<< FAILURES!
[...truncated 447 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1253 tests, 1 failure, 13 ignored (7 assumptions)
Total time: 36 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_25 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_25) - Build #
6252 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6252/
Java: 32bit/jdk1.7.0_25 -server -XX:+UseConcMarkSweepGC
1 tests failed.
FAILED: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 305 from http://127.0.0.1:54846/collection1lastClient and got 247 from http://127.0.0.1:41066/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:54846/collection1lastClient and got 247 from http://127.0.0.1:41066/collection1
at __randomizedtesting.SeedInfo.seed([CB968363466C6444:4A700D7B31330478]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9436 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4] 2> 359979 T1331 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 359985 T1331 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1371889555429
[junit4:junit4] 2> 359985 T1331 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 359986 T1332 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 360086 T1331 oasc.ZkTestServer.run start zk server on port:42857
[junit4:junit4] 2> 360087 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 360230 T1338 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8357d0 name:ZooKeeperConnection Watcher:127.0.0.1:42857 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 360231 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 360231 T1331 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 360235 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 360237 T1340 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18aba7c name:ZooKeeperConnection Watcher:127.0.0.1:42857/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 360237 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 360238 T1331 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 360240 T1331 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 360242 T1331 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 360244 T1331 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 360246 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 360247 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 360253 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 360253 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 360256 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 360256 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 360259 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 360259 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 360261 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 360261 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 360264 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 360264 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 360266 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 360266 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 360268 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 360269 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 360271 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 360271 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 360364 T1331 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 360368 T1331 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57702
[junit4:junit4] 2> 360369 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 360369 T1331 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 360369 T1331 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718
[junit4:junit4] 2> 360370 T1331 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718/solr.xml
[junit4:junit4] 2> 360370 T1331 oasc.CoreContainer.<init> New CoreContainer 24427082
[junit4:junit4] 2> 360371 T1331 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718/'
[junit4:junit4] 2> 360371 T1331 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718/'
[junit4:junit4] 2> 360439 T1331 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 360440 T1331 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 360440 T1331 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 360440 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 360441 T1331 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 360441 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 360442 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 360442 T1331 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 360442 T1331 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 360443 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 360446 T1331 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 360447 T1331 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42857/solr
[junit4:junit4] 2> 360447 T1331 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 360448 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 360449 T1351 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b280a name:ZooKeeperConnection Watcher:127.0.0.1:42857 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 360450 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 360451 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 360454 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 360455 T1353 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d75942 name:ZooKeeperConnection Watcher:127.0.0.1:42857/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 360455 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 360457 T1331 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 360467 T1331 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 360470 T1331 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 360472 T1331 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57702_
[junit4:junit4] 2> 360472 T1331 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57702_
[junit4:junit4] 2> 360477 T1331 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 360482 T1331 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 360484 T1331 oasc.Overseer.start Overseer (id=89908153904726019-127.0.0.1:57702_-n_0000000000) starting
[junit4:junit4] 2> 360488 T1331 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 360493 T1355 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 360493 T1331 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 360503 T1331 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 360505 T1331 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 360508 T1354 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 360510 T1356 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 360510 T1356 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 362012 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 362013 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:57702_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57702"}
[junit4:junit4] 2> 362013 T1354 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 362013 T1354 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 362016 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 362511 T1356 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718/collection1
[junit4:junit4] 2> 362511 T1356 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 362512 T1356 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 362512 T1356 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 362514 T1356 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718/collection1/'
[junit4:junit4] 2> 362514 T1356 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718/collection1/lib/README' to classloader
[junit4:junit4] 2> 362514 T1356 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 362553 T1356 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 362584 T1356 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 362586 T1356 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 362590 T1356 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 362978 T1356 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 362985 T1356 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 362988 T1356 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 363004 T1356 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 363007 T1356 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 363009 T1356 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 363010 T1356 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 363010 T1356 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 363010 T1356 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 363011 T1356 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 363011 T1356 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 363012 T1356 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 363018 T1356 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/
[junit4:junit4] 2> 363018 T1356 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd9f3d
[junit4:junit4] 2> 363018 T1356 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 363018 T1356 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data
[junit4:junit4] 2> 363019 T1356 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index/
[junit4:junit4] 2> 363019 T1356 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 363019 T1356 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index
[junit4:junit4] 2> 363022 T1356 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 363022 T1356 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 363023 T1356 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 363024 T1356 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 363024 T1356 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 363024 T1356 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 363025 T1356 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 363025 T1356 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 363025 T1356 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 363025 T1356 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 363026 T1356 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 363028 T1356 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 363031 T1356 oass.SolrIndexSearcher.<init> Opening Searcher@d4ee71 main
[junit4:junit4] 2> 363032 T1356 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 363032 T1356 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 363035 T1357 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d4ee71 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 363035 T1356 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 363035 T1356 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57702 collection:control_collection shard:shard1
[junit4:junit4] 2> 363036 T1356 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 363041 T1356 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 363043 T1356 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 363044 T1356 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 363044 T1356 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57702/collection1/
[junit4:junit4] 2> 363044 T1356 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 363044 T1356 oasc.SyncStrategy.syncToMe http://127.0.0.1:57702/collection1/ has no replicas
[junit4:junit4] 2> 363044 T1356 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57702/collection1/
[junit4:junit4] 2> 363044 T1356 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 363520 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 363526 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 363550 T1356 oasc.ZkController.register We are http://127.0.0.1:57702/collection1/ and leader is http://127.0.0.1:57702/collection1/
[junit4:junit4] 2> 363550 T1356 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57702
[junit4:junit4] 2> 363551 T1356 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 363551 T1356 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 363551 T1356 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 363553 T1356 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 363554 T1331 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 363554 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 363555 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 363559 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 363560 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 363561 T1360 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fe720e name:ZooKeeperConnection Watcher:127.0.0.1:42857/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 363561 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 363563 T1331 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 363565 T1331 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 363638 T1331 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 363640 T1331 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54846
[junit4:junit4] 2> 363640 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 363640 T1331 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 363641 T1331 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010
[junit4:junit4] 2> 363641 T1331 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010/solr.xml
[junit4:junit4] 2> 363641 T1331 oasc.CoreContainer.<init> New CoreContainer 14823272
[junit4:junit4] 2> 363642 T1331 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010/'
[junit4:junit4] 2> 363642 T1331 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010/'
[junit4:junit4] 2> 363698 T1331 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 363698 T1331 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 363699 T1331 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 363699 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 363699 T1331 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 363699 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 363700 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 363700 T1331 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 363700 T1331 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 363701 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 363703 T1331 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 363704 T1331 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42857/solr
[junit4:junit4] 2> 363704 T1331 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 363705 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 363706 T1371 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d6d6ca name:ZooKeeperConnection Watcher:127.0.0.1:42857 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 363706 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 363708 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 363711 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 363713 T1373 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ef514f name:ZooKeeperConnection Watcher:127.0.0.1:42857/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 363713 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 363717 T1331 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 364720 T1331 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54846_
[junit4:junit4] 2> 364721 T1331 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54846_
[junit4:junit4] 2> 364723 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 364724 T1360 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 364724 T1373 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 364725 T1353 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 364728 T1374 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 364729 T1374 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 365032 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 365033 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:57702__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:57702_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57702"}
[junit4:junit4] 2> 365037 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:54846_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54846"}
[junit4:junit4] 2> 365037 T1354 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 365037 T1354 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 365041 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 365041 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 365041 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 365730 T1374 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010/collection1
[junit4:junit4] 2> 365730 T1374 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 365731 T1374 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 365731 T1374 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 365732 T1374 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010/collection1/'
[junit4:junit4] 2> 365733 T1374 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010/collection1/lib/README' to classloader
[junit4:junit4] 2> 365733 T1374 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 365772 T1374 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 365817 T1374 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 365819 T1374 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 365825 T1374 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 366196 T1374 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 366202 T1374 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 366205 T1374 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 366217 T1374 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 366220 T1374 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 366222 T1374 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 366223 T1374 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 366223 T1374 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 366224 T1374 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 366225 T1374 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 366225 T1374 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 366225 T1374 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 366225 T1374 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/
[junit4:junit4] 2> 366225 T1374 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd9f3d
[junit4:junit4] 2> 366226 T1374 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 366226 T1374 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1
[junit4:junit4] 2> 366226 T1374 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index/
[junit4:junit4] 2> 366226 T1374 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 366227 T1374 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index
[junit4:junit4] 2> 366229 T1374 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 366229 T1374 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 366231 T1374 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 366231 T1374 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 366232 T1374 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 366232 T1374 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 366233 T1374 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 366233 T1374 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 366233 T1374 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 366233 T1374 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 366234 T1374 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 366236 T1374 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 366249 T1374 oass.SolrIndexSearcher.<init> Opening Searcher@1d9ec5a main
[junit4:junit4] 2> 366250 T1374 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 366250 T1374 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 366254 T1375 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d9ec5a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 366255 T1374 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 366255 T1374 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54846 collection:collection1 shard:shard1
[junit4:junit4] 2> 366256 T1374 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 366263 T1374 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 366266 T1374 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 366266 T1374 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 366266 T1374 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54846/collection1/
[junit4:junit4] 2> 366266 T1374 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 366266 T1374 oasc.SyncStrategy.syncToMe http://127.0.0.1:54846/collection1/ has no replicas
[junit4:junit4] 2> 366267 T1374 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54846/collection1/
[junit4:junit4] 2> 366267 T1374 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 366546 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 366553 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 366553 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 366553 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 366572 T1374 oasc.ZkController.register We are http://127.0.0.1:54846/collection1/ and leader is http://127.0.0.1:54846/collection1/
[junit4:junit4] 2> 366573 T1374 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54846
[junit4:junit4] 2> 366573 T1374 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 366573 T1374 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 366573 T1374 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 366576 T1374 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 366577 T1331 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 366577 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 366578 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 366641 T1331 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 366643 T1331 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35789
[junit4:junit4] 2> 366643 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 366644 T1331 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 366644 T1331 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024
[junit4:junit4] 2> 366644 T1331 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024/solr.xml
[junit4:junit4] 2> 366644 T1331 oasc.CoreContainer.<init> New CoreContainer 17288485
[junit4:junit4] 2> 366645 T1331 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024/'
[junit4:junit4] 2> 366645 T1331 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024/'
[junit4:junit4] 2> 366690 T1331 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 366690 T1331 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 366691 T1331 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 366691 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 366691 T1331 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 366691 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 366692 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 366692 T1331 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 366692 T1331 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 366692 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 366695 T1331 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 366695 T1331 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42857/solr
[junit4:junit4] 2> 366696 T1331 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 366696 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 366698 T1387 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cf2cdb name:ZooKeeperConnection Watcher:127.0.0.1:42857 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 366698 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 366700 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 366702 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 366704 T1389 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1eeb9c6 name:ZooKeeperConnection Watcher:127.0.0.1:42857/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 366704 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 366707 T1331 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 367710 T1331 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35789_
[junit4:junit4] 2> 367712 T1331 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35789_
[junit4:junit4] 2> 367714 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 367714 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 367715 T1360 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 367715 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 367715 T1389 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 367716 T1353 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 367716 T1373 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 367724 T1390 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 367724 T1390 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 368058 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 368059 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:54846__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:54846_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54846"}
[junit4:junit4] 2> 368062 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35789_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35789"}
[junit4:junit4] 2> 368062 T1354 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 368062 T1354 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 368068 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 368068 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 368068 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 368071 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 368726 T1390 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024/collection1
[junit4:junit4] 2> 368726 T1390 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 368727 T1390 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 368727 T1390 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 368728 T1390 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024/collection1/'
[junit4:junit4] 2> 368729 T1390 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024/collection1/lib/README' to classloader
[junit4:junit4] 2> 368729 T1390 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 368762 T1390 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 368798 T1390 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 368800 T1390 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 368804 T1390 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 369155 T1390 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 369161 T1390 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 369163 T1390 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 369174 T1390 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 369177 T1390 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 369180 T1390 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 369181 T1390 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 369181 T1390 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 369181 T1390 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 369182 T1390 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 369182 T1390 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 369182 T1390 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 369183 T1390 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/
[junit4:junit4] 2> 369183 T1390 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd9f3d
[junit4:junit4] 2> 369183 T1390 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 369183 T1390 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2
[junit4:junit4] 2> 369184 T1390 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index/
[junit4:junit4] 2> 369184 T1390 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 369184 T1390 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index
[junit4:junit4] 2> 369187 T1390 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 369187 T1390 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 369189 T1390 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 369189 T1390 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 369190 T1390 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 369190 T1390 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 369190 T1390 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 369191 T1390 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 369191 T1390 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 369191 T1390 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 369191 T1390 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 369194 T1390 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 369196 T1390 oass.SolrIndexSearcher.<init> Opening Searcher@169b6c0 main
[junit4:junit4] 2> 369197 T1390 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 369197 T1390 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 369200 T1391 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@169b6c0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 369201 T1390 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 369202 T1390 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35789 collection:collection1 shard:shard1
[junit4:junit4] 2> 369205 T1390 oasc.ZkController.register We are http://127.0.0.1:35789/collection1/ and leader is http://127.0.0.1:54846/collection1/
[junit4:junit4] 2> 369205 T1390 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35789
[junit4:junit4] 2> 369206 T1390 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 369206 T1390 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C4795 name=collection1 org.apache.solr.core.SolrCore@6fd785 url=http://127.0.0.1:35789/collection1 node=127.0.0.1:35789_ C4795_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:35789_, base_url=http://127.0.0.1:35789}
[junit4:junit4] 2> 369206 T1392 C4795 P35789 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 369206 T1392 C4795 P35789 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 369207 T1390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 369207 T1392 C4795 P35789 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 369207 T1392 C4795 P35789 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 369207 T1331 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 369208 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 369208 T1392 C4795 P35789 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 369208 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 369212 T1367 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:35789__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 369274 T1331 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 369276 T1331 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41066
[junit4:junit4] 2> 369277 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 369277 T1331 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 369277 T1331 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655
[junit4:junit4] 2> 369277 T1331 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655/solr.xml
[junit4:junit4] 2> 369278 T1331 oasc.CoreContainer.<init> New CoreContainer 29850143
[junit4:junit4] 2> 369278 T1331 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655/'
[junit4:junit4] 2> 369278 T1331 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655/'
[junit4:junit4] 2> 369321 T1331 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 369321 T1331 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 369321 T1331 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 369322 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 369322 T1331 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 369322 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 369322 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 369323 T1331 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 369323 T1331 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 369323 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 369326 T1331 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 369326 T1331 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42857/solr
[junit4:junit4] 2> 369326 T1331 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 369327 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 369329 T1404 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54c80c name:ZooKeeperConnection Watcher:127.0.0.1:42857 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 369330 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 369332 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 369334 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 369335 T1406 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f53d3f name:ZooKeeperConnection Watcher:127.0.0.1:42857/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 369336 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 369339 T1331 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 369575 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 369576 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:35789__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35789_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35789"}
[junit4:junit4] 2> 369579 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 369579 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 369579 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 369579 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 369580 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 370212 T1367 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:35789__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 370212 T1367 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:35789__collection1&state=recovering&nodeName=127.0.0.1:35789_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1000
[junit4:junit4] 2> 370341 T1331 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41066_
[junit4:junit4] 2> 370342 T1331 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41066_
[junit4:junit4] 2> 370344 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 370345 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 370345 T1360 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 370344 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 370345 T1406 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 370345 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 370345 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 370346 T1353 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 370347 T1389 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 370347 T1373 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 370351 T1407 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 370351 T1407 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 371084 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 371085 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41066_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41066"}
[junit4:junit4] 2> 371085 T1354 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 371085 T1354 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 371089 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 371089 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 371089 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 371089 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 371089 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 371353 T1407 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655/collection1
[junit4:junit4] 2> 371353 T1407 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 371354 T1407 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 371354 T1407 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 371355 T1407 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655/collection1/'
[junit4:junit4] 2> 371355 T1407 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655/collection1/lib/README' to classloader
[junit4:junit4] 2> 371356 T1407 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 371386 T1407 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 371428 T1407 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 371429 T1407 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 371434 T1407 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 371798 T1407 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 371807 T1407 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 371810 T1407 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 371822 T1407 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 371826 T1407 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 371830 T1407 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 371831 T1407 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 371831 T1407 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 371831 T1407 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 371832 T1407 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 371832 T1407 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 371832 T1407 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 371832 T1407 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/
[junit4:junit4] 2> 371833 T1407 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd9f3d
[junit4:junit4] 2> 371833 T1407 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 371833 T1407 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3
[junit4:junit4] 2> 371833 T1407 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index/
[junit4:junit4] 2> 371834 T1407 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 371834 T1407 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index
[junit4:junit4] 2> 371836 T1407 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 371837 T1407 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 371838 T1407 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 371838 T1407 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 371839 T1407 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 371839 T1407 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 371840 T1407 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 371840 T1407 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 371840 T1407 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 371841 T1407 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 371841 T1407 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 371843 T1407 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 371845 T1407 oass.SolrIndexSearcher.<init> Opening Searcher@13819d3 main
[junit4:junit4] 2> 371846 T1407 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 371846 T1407 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 371850 T1408 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13819d3 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 371852 T1407 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 371852 T1407 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41066 collection:collection1 shard:shard1
[junit4:junit4] 2> 371855 T1407 oasc.ZkController.register We are http://127.0.0.1:41066/collection1/ and leader is http://127.0.0.1:54846/collection1/
[junit4:junit4] 2> 371855 T1407 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41066
[junit4:junit4] 2> 371855 T1407 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 371855 T1407 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C4796 name=collection1 org.apache.solr.core.SolrCore@43017e url=http://127.0.0.1:41066/collection1 node=127.0.0.1:41066_ C4796_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:41066_, base_url=http://127.0.0.1:41066}
[junit4:junit4] 2> 371856 T1409 C4796 P41066 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 371856 T1409 C4796 P41066 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 371856 T1407 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 371856 T1409 C4796 P41066 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 371857 T1409 C4796 P41066 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 371857 T1331 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 371858 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 371858 T1409 C4796 P41066 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 371858 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 371862 T1366 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:41066__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 371921 T1331 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 371923 T1331 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37613
[junit4:junit4] 2> 371923 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 371923 T1331 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 371924 T1331 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304
[junit4:junit4] 2> 371924 T1331 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304/solr.xml
[junit4:junit4] 2> 371924 T1331 oasc.CoreContainer.<init> New CoreContainer 12986159
[junit4:junit4] 2> 371924 T1331 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304/'
[junit4:junit4] 2> 371925 T1331 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304/'
[junit4:junit4] 2> 371970 T1331 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 371971 T1331 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 371971 T1331 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 371971 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 371971 T1331 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 371972 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 371972 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 371972 T1331 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 371972 T1331 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 371973 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 371975 T1331 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 371976 T1331 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42857/solr
[junit4:junit4] 2> 371976 T1331 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 371977 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 371978 T1421 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46619f name:ZooKeeperConnection Watcher:127.0.0.1:42857 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 371979 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 371981 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 371983 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 371985 T1423 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@df0925 name:ZooKeeperConnection Watcher:127.0.0.1:42857/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 371985 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 371989 T1331 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C4797 name=collection1 org.apache.solr.core.SolrCore@6fd785 url=http://127.0.0.1:35789/collection1 node=127.0.0.1:35789_ C4797_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:35789_, base_url=http://127.0.0.1:35789}
[junit4:junit4] 2> 372213 T1392 C4797 P35789 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:54846/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 372213 T1392 C4797 P35789 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:35789 START replicas=[http://127.0.0.1:54846/collection1/] nUpdates=100
[junit4:junit4] 2> 372214 T1392 C4797 P35789 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 372214 T1392 C4797 P35789 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 372214 T1392 C4797 P35789 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 372214 T1392 C4797 P35789 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 372214 T1392 C4797 P35789 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 372215 T1392 C4797 P35789 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:54846/collection1/. core=collection1
[junit4:junit4] 2> 372215 T1392 C4797 P35789 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C4798 name=collection1 org.apache.solr.core.SolrCore@1df5130 url=http://127.0.0.1:54846/collection1 node=127.0.0.1:54846_ C4798_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:54846_, base_url=http://127.0.0.1:54846, leader=true}
[junit4:junit4] 2> 372219 T1368 C4798 P54846 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 372221 T1367 C4798 P54846 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 372223 T1367 C4798 P54846 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 372223 T1367 C4798 P54846 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 372230 T1367 C4798 P54846 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 372230 T1367 C4798 P54846 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 372231 T1367 C4798 P54846 oass.SolrIndexSearcher.<init> Opening Searcher@1ece40a realtime
[junit4:junit4] 2> 372231 T1367 C4798 P54846 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 372231 T1367 C4798 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 10
[junit4:junit4] 2> 372232 T1392 C4797 P35789 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 372232 T1392 C4797 P35789 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 372233 T1368 C4798 P54846 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 372234 T1368 C4798 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 372234 T1392 C4797 P35789 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 372234 T1392 C4797 P35789 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 372234 T1392 C4797 P35789 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 372236 T1367 C4798 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 372237 T1392 C4797 P35789 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 372237 T1392 C4797 P35789 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index.20130622172607681
[junit4:junit4] 2> 372238 T1392 C4797 P35789 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index.20130622172607681 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e49963 fullCopy=false
[junit4:junit4] 2> 372240 T1369 C4798 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 372241 T1392 C4797 P35789 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 372250 T1392 C4797 P35789 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 372250 T1392 C4797 P35789 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 372252 T1392 C4797 P35789 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 372252 T1392 C4797 P35789 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 372252 T1392 C4797 P35789 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 372252 T1392 C4797 P35789 oass.SolrIndexSearcher.<init> Opening Searcher@115e129 main
[junit4:junit4] 2> 372253 T1391 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@115e129 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 372254 T1392 C4797 P35789 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index.20130622172607681 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index.20130622172607681;done=true>>]
[junit4:junit4] 2> 372254 T1392 C4797 P35789 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index.20130622172607681
[junit4:junit4] 2> 372254 T1392 C4797 P35789 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index.20130622172607681
[junit4:junit4] 2> 372254 T1392 C4797 P35789 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 372254 T1392 C4797 P35789 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 372254 T1392 C4797 P35789 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 372255 T1392 C4797 P35789 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 372256 T1392 C4797 P35789 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 372594 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 372595 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:41066__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41066_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41066"}
[junit4:junit4] 2> 372599 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:35789__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:35789_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:35789"}
[junit4:junit4] 2> 372620 T1423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 372620 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 372620 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 372620 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 372620 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 372620 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 372862 T1366 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:41066__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 372863 T1366 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:41066__collection1&state=recovering&nodeName=127.0.0.1:41066_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4] 2> 372992 T1331 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37613_
[junit4:junit4] 2> 372993 T1331 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37613_
[junit4:junit4] 2> 372995 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 372995 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 372995 T1360 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 372996 T1406 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 372996 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 372996 T1423 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 372997 T1353 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 372995 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 372997 T1423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 372997 T1373 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 372996 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 372999 T1389 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 373004 T1426 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 373004 T1426 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 374131 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 374132 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:37613_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:37613"}
[junit4:junit4] 2> 374132 T1354 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 374132 T1354 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 374136 T1423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 374136 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 374136 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 374136 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 374136 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 374136 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> ASYNC NEW_CORE C4799 name=collection1 org.apache.solr.core.SolrCore@43017e url=http://127.0.0.1:41066/collection1 node=127.0.0.1:41066_ C4799_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:41066_, base_url=http://127.0.0.1:41066}
[junit4:junit4] 2> 374863 T1409 C4799 P41066 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:54846/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 374864 T1409 C4799 P41066 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:41066 START replicas=[http://127.0.0.1:54846/collection1/] nUpdates=100
[junit4:junit4] 2> 374864 T1409 C4799 P41066 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 374864 T1409 C4799 P41066 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 374864 T1409 C4799 P41066 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 374865 T1409 C4799 P41066 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 374865 T1409 C4799 P41066 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 374865 T1409 C4799 P41066 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:54846/collection1/. core=collection1
[junit4:junit4] 2> 374865 T1409 C4799 P41066 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C4800 name=collection1 org.apache.solr.core.SolrCore@1df5130 url=http://127.0.0.1:54846/collection1 node=127.0.0.1:54846_ C4800_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:54846_, base_url=http://127.0.0.1:54846, leader=true}
[junit4:junit4] 2> 374867 T1369 C4800 P54846 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 374869 T1368 C4800 P54846 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 374880 T1368 C4800 P54846 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> 374880 T1368 C4800 P54846 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 374880 T1368 C4800 P54846 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 374881 T1368 C4800 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 12
[junit4:junit4] 2> 374881 T1409 C4799 P41066 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 374881 T1409 C4799 P41066 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 374884 T1366 C4800 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 374884 T1409 C4799 P41066 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4] 2> 374884 T1409 C4799 P41066 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 374884 T1409 C4799 P41066 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 374886 T1369 C4800 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=3&version=2} status=0 QTime=0
[junit4:junit4] 2> 374887 T1409 C4799 P41066 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 374887 T1409 C4799 P41066 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index.20130622172610331
[junit4:junit4] 2> 374887 T1409 C4799 P41066 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index.20130622172610331 lockFactory=org.apache.lucene.store.NativeFSLockFactory@19b2abb fullCopy=false
[junit4:junit4] 2> 374890 T1368 C4800 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_3&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=3} status=0 QTime=0
[junit4:junit4] 2> 374891 T1409 C4799 P41066 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 374893 T1409 C4799 P41066 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 374893 T1409 C4799 P41066 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 374894 T1409 C4799 P41066 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> 374894 T1409 C4799 P41066 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 374895 T1409 C4799 P41066 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 374895 T1409 C4799 P41066 oass.SolrIndexSearcher.<init> Opening Searcher@19d6ea2 main
[junit4:junit4] 2> 374896 T1408 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19d6ea2 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 374896 T1409 C4799 P41066 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index.20130622172610331 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index.20130622172610331;done=true>>]
[junit4:junit4] 2> 374896 T1409 C4799 P41066 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index.20130622172610331
[junit4:junit4] 2> 374896 T1409 C4799 P41066 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index.20130622172610331
[junit4:junit4] 2> 374896 T1409 C4799 P41066 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 374897 T1409 C4799 P41066 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 374897 T1409 C4799 P41066 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 374897 T1409 C4799 P41066 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 374910 T1409 C4799 P41066 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 375007 T1426 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304/collection1
[junit4:junit4] 2> 375007 T1426 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 375008 T1426 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 375008 T1426 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 375009 T1426 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304/collection1/'
[junit4:junit4] 2> 375009 T1426 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304/collection1/lib/README' to classloader
[junit4:junit4] 2> 375010 T1426 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 375041 T1426 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 375072 T1426 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 375074 T1426 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 375077 T1426 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 375461 T1426 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 375467 T1426 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 375470 T1426 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 375480 T1426 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 375483 T1426 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 375489 T1426 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 375490 T1426 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 375490 T1426 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 375490 T1426 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 375491 T1426 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 375491 T1426 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 375491 T1426 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 375492 T1426 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/
[junit4:junit4] 2> 375492 T1426 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd9f3d
[junit4:junit4] 2> 375492 T1426 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 375493 T1426 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4
[junit4:junit4] 2> 375493 T1426 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index/
[junit4:junit4] 2> 375493 T1426 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 375494 T1426 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index
[junit4:junit4] 2> 375531 T1426 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 375532 T1426 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 375533 T1426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 375533 T1426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 375534 T1426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 375534 T1426 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 375535 T1426 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 375535 T1426 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 375536 T1426 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 375536 T1426 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 375537 T1426 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 375539 T1426 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 375541 T1426 oass.SolrIndexSearcher.<init> Opening Searcher@1e410ff main
[junit4:junit4] 2> 375542 T1426 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 375542 T1426 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 375546 T1429 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e410ff main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 375548 T1426 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 375549 T1426 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37613 collection:collection1 shard:shard1
[junit4:junit4] 2> 375552 T1426 oasc.ZkController.register We are http://127.0.0.1:37613/collection1/ and leader is http://127.0.0.1:54846/collection1/
[junit4:junit4] 2> 375552 T1426 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37613
[junit4:junit4] 2> 375552 T1426 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 375553 T1426 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C4801 name=collection1 org.apache.solr.core.SolrCore@1a9d4fc url=http://127.0.0.1:37613/collection1 node=127.0.0.1:37613_ C4801_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:37613_, base_url=http://127.0.0.1:37613}
[junit4:junit4] 2> 375553 T1430 C4801 P37613 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 375553 T1430 C4801 P37613 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 375554 T1430 C4801 P37613 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 375554 T1426 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 375554 T1430 C4801 P37613 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 375555 T1331 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 375555 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 375555 T1430 C4801 P37613 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 375556 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 375559 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 375560 T1368 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:37613__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 375560 T1331 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 375560 T1331 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 375561 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 375641 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 375642 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:41066__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41066_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41066"}
[junit4:junit4] 2> 375644 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:37613__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:37613_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:37613"}
[junit4:junit4] 2> 375647 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 375648 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 375648 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 375648 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 375647 T1423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 375649 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 376560 T1368 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:37613__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 376560 T1368 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:37613__collection1&state=recovering&nodeName=127.0.0.1:37613_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 376563 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 377564 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 377755 T3 oasc.CoreContainer.finalize ERROR CoreContainer was not shutdown prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!! instance=31836597
[junit4:junit4] 2> C4801_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:37613_, base_url=http://127.0.0.1:37613}
[junit4:junit4] 2> 378561 T1430 C4801 P37613 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:54846/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 378561 T1430 C4801 P37613 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:37613 START replicas=[http://127.0.0.1:54846/collection1/] nUpdates=100
[junit4:junit4] 2> 378562 T1430 C4801 P37613 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 378562 T1430 C4801 P37613 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 378562 T1430 C4801 P37613 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 378562 T1430 C4801 P37613 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 378562 T1430 C4801 P37613 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 378563 T1430 C4801 P37613 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:54846/collection1/. core=collection1
[junit4:junit4] 2> 378563 T1430 C4801 P37613 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C4802 name=collection1 org.apache.solr.core.SolrCore@1df5130 url=http://127.0.0.1:54846/collection1 node=127.0.0.1:54846_ C4802_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:54846_, base_url=http://127.0.0.1:54846, leader=true}
[junit4:junit4] 2> 378564 T1369 C4802 P54846 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 378566 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 378568 T1368 C4802 P54846 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 378581 T1368 C4802 P54846 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> 378581 T1368 C4802 P54846 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 378582 T1368 C4802 P54846 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 378582 T1368 C4802 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 14
[junit4:junit4] 2> 378583 T1430 C4801 P37613 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 378584 T1430 C4801 P37613 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 378586 T1367 C4802 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 378586 T1430 C4801 P37613 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4] 2> 378587 T1430 C4801 P37613 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 378587 T1430 C4801 P37613 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 378589 T1366 C4802 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=4&version=2} status=0 QTime=0
[junit4:junit4] 2> 378590 T1430 C4801 P37613 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 378591 T1430 C4801 P37613 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index.20130622172614034
[junit4:junit4] 2> 378591 T1430 C4801 P37613 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index.20130622172614034 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c400a fullCopy=false
[junit4:junit4] 2> 378594 T1368 C4802 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_4&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=4} status=0 QTime=0
[junit4:junit4] 2> 378595 T1430 C4801 P37613 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 378597 T1430 C4801 P37613 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 378598 T1430 C4801 P37613 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 378600 T1430 C4801 P37613 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> 378600 T1430 C4801 P37613 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 378600 T1430 C4801 P37613 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 378601 T1430 C4801 P37613 oass.SolrIndexSearcher.<init> Opening Searcher@48157b main
[junit4:junit4] 2> 378602 T1429 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@48157b main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 378607 T1430 C4801 P37613 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index.20130622172614034 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index.20130622172614034;done=true>>]
[junit4:junit4] 2> 378608 T1430 C4801 P37613 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index.20130622172614034
[junit4:junit4] 2> 378608 T1430 C4801 P37613 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index.20130622172614034
[junit4:junit4] 2> 378608 T1430 C4801 P37613 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 378608 T1430 C4801 P37613 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 378608 T1430 C4801 P37613 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 378608 T1430 C4801 P37613 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 378611 T1430 C4801 P37613 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 378657 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 378658 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:37613__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:37613_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:37613"}
[junit4:junit4] 2> 378661 T1423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 378662 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 378662 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 378662 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 378662 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 378662 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 379568 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 379569 T1331 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> ASYNC NEW_CORE C4803 name=collection1 org.apache.solr.core.SolrCore@18c7b8d url=http://127.0.0.1:57702/collection1 node=127.0.0.1:57702_ C4803_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:57702_, base_url=http://127.0.0.1:57702, leader=true}
[junit4:junit4] 2> 379575 T1348 C4803 P57702 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 379577 T1348 C4803 P57702 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 379577 T1348 C4803 P57702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 379650 T1348 C4803 P57702 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 379650 T1348 C4803 P57702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 379651 T1348 C4803 P57702 oass.SolrIndexSearcher.<init> Opening Searcher@13423d0 main
[junit4:junit4] 2> 379651 T1348 C4803 P57702 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 379655 T1357 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13423d0 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> ASYNC NEW_CORE C4804 name=collection1 org.apache.solr.core.SolrCore@18c7b8d url=http://127.0.0.1:57702/collection1 node=127.0.0.1:57702_ C4804_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:57702_, base_url=http://127.0.0.1:57702, leader=true}
[junit4:junit4] 2> 379656 T1348 C4804 P57702 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 82
[junit4:junit4] 2> ASYNC NEW_CORE C4805 name=collection1 org.apache.solr.core.SolrCore@6fd785 url=http://127.0.0.1:35789/collection1 node=127.0.0.1:35789_ C4805_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:35789_, base_url=http://127.0.0.1:35789}
[junit4:junit4] 2> 379666 T1382 C4805 P35789 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:54846/collection1/, StdNode: http://127.0.0.1:35789/collection1/, StdNode: http://127.0.0.1:41066/collection1/, StdNode: http://127.0.0.1:37613/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> ASYNC NEW_CORE C4806 name=collection1 org.apache.solr.core.SolrCore@1df5130 url=http://127.0.0.1:54846/collection1 node=127.0.0.1:54846_ C4806_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:54846_, base_url=http://127.0.0.1:54846, leader=true}
[junit4:junit4] 2> 379669 T1368 C4806 P54846 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 379671 T1385 C4805 P35789 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C4807 name=collection1 org.apache.solr.core.SolrCore@1a9d4fc url=http://127.0.0.1:37613/collection1 node=127.0.0.1:37613_ C4807_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:37613_, base_url=http://127.0.0.1:37613}
[junit4:junit4] 2> 379673 T1419 C4807 P37613 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C4808 name=collection1 org.apache.solr.core.SolrCore@43017e url=http://127.0.0.1:41066/collection1 node=127.0.0.1:41066_ C4808_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:41066_, base_url=http://127.0.0.1:41066}
[junit4:junit4] 2> 379676 T1399 C4808 P41066 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 379690 T1368 C4806 P54846 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=4
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> 379690 T1368 C4806 P54846 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 379691 T1368 C4806 P54846 oass.SolrIndexSearcher.<init> Opening Searcher@13ba920 main
[junit4:junit4] 2> 379691 T1368 C4806 P54846 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 379692 T1375 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13ba920 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 379692 T1368 C4806 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 23
[junit4:junit4] 2> 379720 T1419 C4807 P37613 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> 379720 T1399 C4808 P41066 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> 379720 T1385 C4805 P35789 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> 379721 T1399 C4808 P41066 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 379720 T1419 C4807 P37613 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 379721 T1385 C4805 P35789 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 379722 T1399 C4808 P41066 oass.SolrIndexSearcher.<init> Opening Searcher@1d8bc2d main
[junit4:junit4] 2> 379723 T1419 C4807 P37613 oass.SolrIndexSearcher.<init> Opening Searcher@ed35dd main
[junit4:junit4] 2> 379723 T1385 C4805 P35789 oass.SolrIndexSearcher.<init> Opening Searcher@2a9f0e main
[junit4:junit4] 2> 379723 T1399 C4808 P41066 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 379723 T1408 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d8bc2d main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4] 2> 379723 T1385 C4805 P35789 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 379723 T1419 C4807 P37613 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 379724 T1399 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 48
[junit4:junit4] 2> 379724 T1391 oasc.JmxMonitoredMap.put WARN Failed to register info bean: fieldValueCache javax.management.InstanceAlreadyExistsException: solr/collection1:type=fieldValueCache,id=org.apache.solr.search.FastLRUCache
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4] 2> at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:303)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1846)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1723)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2>
[junit4:junit4] 2> 379724 T1429 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ed35dd main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4] 2> 379725 T1391 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2a9f0e main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 379726 T1419 C4807 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 53
[junit4:junit4] 2> 379726 T1385 C4805 P35789 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 55
[junit4:junit4] 2> 379727 T1382 C4805 P35789 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 61
[junit4:junit4] 2> 379728 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 379730 T1367 C4806 P54846 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 379731 T1384 C4805 P35789 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 379733 T1400 C4808 P41066 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 379734 T1417 C4807 P37613 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4] 2> 381738 T1347 C4804 P57702 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1438530485282144256)} 0 1
[junit4:junit4] 2> 381744 T1382 C4805 P35789 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1438530485286338560&update.from=http://127.0.0.1:54846/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438530485286338560)} 0 1
[junit4:junit4] 2> 381744 T1419 C4807 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1438530485286338560&update.from=http://127.0.0.1:54846/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438530485286338560)} 0 1
[junit4:junit4] 2> 381746 T1402 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1438530485286338560&update.from=http://127.0.0.1:54846/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438530485286338560)} 0 1
[junit4:junit4] 2> 381747 T1369 C4806 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1438530485286338560)} 0 6
[junit4:junit4] 2> 381747 T1399 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 8
[junit4:junit4] 2> 381751 T1348 C4804 P57702 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1438530485294727168)]} 0 2
[junit4:junit4] 2> 381761 T1417 C4807 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438530485301018624)]} 0 2
[junit4:junit4] 2> 381761 T1384 C4805 P35789 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438530485301018624)]} 0 2
[junit4:junit4] 2> 381763 T1401 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438530485301018624)]} 0 1
[junit4:junit4] 2> 381764 T1367 C4806 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:41066/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1438530485301018624)]} 0 9
[junit4:junit4] 2> 381765 T1400 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0]} 0 12
[junit4:junit4] 2> 381767 T1346 C4804 P57702 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1438530485313601536)]} 0 0
[junit4:junit4] 2> 381772 T1402 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438530485315698688)]} 0 0
[junit4:junit4] 2> 381773 T1383 C4805 P35789 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438530485315698688)]} 0 1
[junit4:junit4] 2> 381773 T1416 C4807 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438530485315698688)]} 0 1
[junit4:junit4] 2> 381774 T1366 C4806 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1438530485315698688)]} 0 5
[junit4:junit4] 2> 381777 T1349 C4804 P57702 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1438530485323038720)]} 0 1
[junit4:junit4] 2> 381780 T1368 C4806 P54846 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:35789/collection1/]
[junit4:junit4] 2> 381781 T1368 C4806 P54846 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:35789/collection1/ against:[http://127.0.0.1:35789/collection1/] result:true
[junit4:junit4] 2> 381781 T1368 C4806 P54846 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:41066/collection1/ against:[http://127.0.0.1:35789/collection1/] result:false
[junit4:junit4] 2> 381781 T1368 C4806 P54846 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:37613/collection1/ against:[http://127.0.0.1:35789/collection1/] result:false
[junit4:junit4] 2> 381785 T1419 C4807 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:35789/collection1/&wt=javabin&version=2} {add=[2 (1438530485328281600)]} 0 1
[junit4:junit4] 2> 381785 T1399 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:35789/collection1/&wt=javabin&version=2} {add=[2 (1438530485328281600)]} 0 1
[junit4:junit4] 2> 381786 T1368 C4806 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:37613/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:35789/collection1/&wt=javabin&version=2} {add=[2 (1438530485328281600)]} 0 6
[junit4:junit4] 2> 381786 T1418 C4807 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:35789/collection1/&wt=javabin&version=2} {add=[2]} 0 7
[junit4:junit4] 2> 381790 T1347 C4804 P57702 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1438530485336670208)]} 0 1
[junit4:junit4] 2> 381795 T1369 C4806 P54846 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:35789/collection1/]
[junit4:junit4] 2> 381795 T1369 C4806 P54846 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:35789/collection1/ against:[http://127.0.0.1:35789/collection1/] result:true
[junit4:junit4] 2> 381796 T1369 C4806 P54846 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:41066/collection1/ against:[http://127.0.0.1:35789/collection1/] result:false
[junit4:junit4] 2> 381796 T1369 C4806 P54846 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:37613/collection1/ against:[http://127.0.0.1:35789/collection1/] result:false
[junit4:junit4] 2> 381800 T1417 C4807 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:35789/collection1/&wt=javabin&version=2} {add=[3 (1438530485344010240)]} 0 1
[junit4:junit4] 2> 381800 T1400 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:35789/collection1/&wt=javabin&version=2} {add=[3 (1438530485344010240)]} 0 1
[junit4:junit4] 2> 381801 T1369 C4806 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:41066/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:35789/collection1/&wt=javabin&version=2} {add=[3 (1438530485344010240)]} 0 6
[junit4:junit4] 2> 381801 T1401 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:35789/collection1/&test.distrib.skip.servers=http://127.0.0.1:41066/collection1/&wt=javabin&version=2} {add=[3]} 0 8
[junit4:junit4] 2> 381803 T1348 C4804 P57702 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C4809 name=collection1 org.apache.solr.core.SolrCore@18c7b8d url=http://127.0.0.1:57702/collection1 node=127.0.0.1:57702_ C4809_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:57702_, base_url=http://127.0.0.1:57702, leader=true}
[junit4:junit4] 2> 381830 T1348 C4809 P57702 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> 381830 T1348 C4809 P57702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4] 2> 381834 T1348 C4809 P57702 oass.SolrIndexSearcher.<init> Opening Searcher@b39bb5 main
[junit4:junit4] 2> 381835 T1348 C4809 P57702 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 381835 T1357 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b39bb5 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 381836 T1348 C4809 P57702 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 33
[junit4:junit4] 2> ASYNC NEW_CORE C4810 name=collection1 org.apache.solr.core.SolrCore@1a9d4fc url=http://127.0.0.1:37613/collection1 node=127.0.0.1:37613_ C4810_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:37613_, base_url=http://127.0.0.1:37613}
[junit4:junit4] 2> 381838 T1416 C4810 P37613 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:54846/collection1/, StdNode: http://127.0.0.1:35789/collection1/, StdNode: http://127.0.0.1:41066/collection1/, StdNode: http://127.0.0.1:37613/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> ASYNC NEW_CORE C4811 name=collection1 org.apache.solr.core.SolrCore@1df5130 url=http://127.0.0.1:54846/collection1 node=127.0.0.1:54846_ C4811_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:54846_, base_url=http://127.0.0.1:54846, leader=true}
[junit4:junit4] 2> 381841 T1367 C4811 P54846 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C4812 name=collection1 org.apache.solr.core.SolrCore@6fd785 url=http://127.0.0.1:35789/collection1 node=127.0.0.1:35789_ C4812_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:35789_, base_url=http://127.0.0.1:35789}
[junit4:junit4] 2> 381841 T1384 C4812 P35789 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 381841 T1418 C4810 P37613 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C4813 name=collection1 org.apache.solr.core.SolrCore@43017e url=http://127.0.0.1:41066/collection1 node=127.0.0.1:41066_ C4813_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:41066_, base_url=http://127.0.0.1:41066}
[junit4:junit4] 2> 381841 T1399 C4813 P41066 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 381900 T1399 C4813 P41066 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> 381900 T1399 C4813 P41066 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4] 2> 381900 T1418 C4810 P37613 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_6,generation=6}
[junit4:junit4] 2> 381901 T1418 C4810 P37613 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 381900 T1367 C4811 P54846 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_6,generation=6}
[junit4:junit4] 2> 381905 T1367 C4811 P54846 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 381905 T1399 C4813 P41066 oass.SolrIndexSearcher.<init> Opening Searcher@499ec1 main
[junit4:junit4] 2> 381905 T1418 C4810 P37613 oass.SolrIndexSearcher.<init> Opening Searcher@1f6659e main
[junit4:junit4] 2> 381906 T1399 C4813 P41066 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 381906 T1429 oasc.JmxMonitoredMap.put WARN Failed to register info bean: searcher javax.management.InstanceAlreadyExistsException: solr/collection1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4] 2> at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:299)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1846)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1723)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2>
[junit4:junit4] 2> 381908 T1429 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f6659e main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 381906 T1418 C4810 P37613 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 381907 T1408 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@499ec1 main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 381909 T1418 C4810 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 68
[junit4:junit4] 2> 381909 T1367 C4811 P54846 oass.SolrIndexSearcher.<init> Opening Searcher@1bf1de3 main
[junit4:junit4] 2> 381910 T1384 C4812 P35789 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> 381910 T1384 C4812 P35789 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 381911 T1375 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1bf1de3 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 381910 T1399 C4813 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 69
[junit4:junit4] 2> 381910 T1367 C4811 P54846 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 381912 T1367 C4811 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 71
[junit4:junit4] 2> 381914 T1384 C4812 P35789 oass.SolrIndexSearcher.<init> Opening Searcher@10d4541 main
[junit4:junit4] 2> 381914 T1384 C4812 P35789 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 381915 T1391 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@10d4541 main{StandardDirectoryReader(segments_4:3:nrt _0(5.0):C2)}
[junit4:junit4] 2> 381915 T1384 C4812 P35789 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 74
[junit4:junit4] 2> 381916 T1416 C4810 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 78
[junit4:junit4] 2> 381916 T1331 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 381917 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 381918 T1331 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 381920 T1366 C4811 P54846 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 381922 T1383 C4812 P35789 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0
[junit4:junit4] 2> 381925 T1400 C4813 P41066 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 381927 T1417 C4810 P37613 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4] 2> 381927 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 381931 T1401 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4] 2> 381932 T1401 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 381936 T1368 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4] 2> 381936 T1368 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54846/collection1/
[junit4:junit4] 2> 381937 T1368 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:54846 START replicas=[http://127.0.0.1:35789/collection1/, http://127.0.0.1:41066/collection1/, http://127.0.0.1:37613/collection1/] nUpdates=100
[junit4:junit4] 2> 381938 T1385 C4812 P35789 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 381939 T1368 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:54846 Received 3 versions from 127.0.0.1:35789/collection1/
[junit4:junit4] 2> 381939 T1419 C4810 P37613 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 381939 T1402 C4813 P41066 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 381939 T1368 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:54846 Our versions are newer. ourLowThreshold=1438530485286338560 otherHigh=1438530485315698688
[junit4:junit4] 2> 381940 T1368 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:54846 Received 5 versions from 127.0.0.1:37613/collection1/
[junit4:junit4] 2> 381940 T1368 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:54846 Our versions are newer. ourLowThreshold=1438530485286338560 otherHigh=1438530485328281600
[junit4:junit4] 2> 381940 T1368 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:54846 Received 5 versions from 127.0.0.1:41066/collection1/
[junit4:junit4] 2> 381941 T1368 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:54846 Our versions are newer. ourLowThreshold=1438530485286338560 otherHigh=1438530485328281600
[junit4:junit4] 2> 381941 T1368 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:54846 DONE. sync succeeded
[junit4:junit4] 2> 381941 T1368 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 381941 T1368 oasc.SyncStrategy.syncToMe http://127.0.0.1:54846/collection1/: try and ask http://127.0.0.1:35789/collection1/ to sync
[junit4:junit4] 2> 381942 T1368 oasc.SyncStrategy.syncToMe http://127.0.0.1:54846/collection1/: try and ask http://127.0.0.1:41066/collection1/ to sync
[junit4:junit4] 2> 381942 T1368 oasc.SyncStrategy.syncToMe http://127.0.0.1:54846/collection1/: try and ask http://127.0.0.1:37613/collection1/ to sync
[junit4:junit4] 2> 381943 T1384 C4812 P35789 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:35789 START replicas=[http://127.0.0.1:54846/collection1/] nUpdates=100
[junit4:junit4] 2> 381943 T1400 C4813 P41066 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:41066 START replicas=[http://127.0.0.1:54846/collection1/] nUpdates=100
[junit4:junit4] 2> 381946 T1418 C4810 P37613 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:37613 START replicas=[http://127.0.0.1:54846/collection1/] nUpdates=100
[junit4:junit4] 2> 381947 T1367 C4811 P54846 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 381947 T1366 C4811 P54846 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 381947 T1400 C4813 P41066 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:41066 Received 5 versions from 127.0.0.1:54846/collection1/
[junit4:junit4] 2> 381948 T1384 C4812 P35789 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:35789 Received 5 versions from 127.0.0.1:54846/collection1/
[junit4:junit4] 2> 381948 T1367 C4811 P54846 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 381948 T1384 C4812 P35789 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:35789 Requesting updates from 127.0.0.1:54846/collection1/n=2 versions=[1438530485344010240, 1438530485328281600]
[junit4:junit4] 2> 381948 T1400 C4813 P41066 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:41066 Our versions are newer. ourLowThreshold=1438530485286338560 otherHigh=1438530485328281600
[junit4:junit4] 2> 381949 T1400 C4813 P41066 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:41066 DONE. sync succeeded
[junit4:junit4] 2> 381949 T1418 C4810 P37613 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:37613 Received 5 versions from 127.0.0.1:54846/collection1/
[junit4:junit4] 2> 381949 T1400 C4813 P41066 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:54846/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=6
[junit4:junit4] 2> 381950 T1418 C4810 P37613 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:37613 Our versions are newer. ourLowThreshold=1438530485286338560 otherHigh=1438530485328281600
[junit4:junit4] 2> 381950 T1418 C4810 P37613 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:37613 DONE. sync succeeded
[junit4:junit4] 2> 381950 T1418 C4810 P37613 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:54846/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=4
[junit4:junit4] 2> 381951 T1368 oasc.SyncStrategy.syncToMe http://127.0.0.1:54846/collection1/: sync completed with http://127.0.0.1:41066/collection1/
[junit4:junit4] 2> 381951 T1366 C4811 P54846 oasc.SolrCore.execute [collection1] webapp= path=/get params={distrib=false&getUpdates=1438530485344010240,1438530485328281600&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 381952 T1368 oasc.SyncStrategy.syncToMe http://127.0.0.1:54846/collection1/: sync completed with http://127.0.0.1:37613/collection1/
[junit4:junit4] 2> 381954 T1384 C4812 P35789 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1438530485328281600), 3 (1438530485344010240)]} 0 2
[junit4:junit4] 2> 381954 T1384 C4812 P35789 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:35789 DONE. sync succeeded
[junit4:junit4] 2> 381955 T1384 C4812 P35789 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:54846/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=12
[junit4:junit4] 2> 381955 T1368 oasc.SyncStrategy.syncToMe http://127.0.0.1:54846/collection1/: sync completed with http://127.0.0.1:35789/collection1/
[junit4:junit4] 2> 381956 T1368 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=20
[junit4:junit4] 2> 381956 T1401 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=25
[junit4:junit4] 2> 381956 T1331 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4] 2> 381957 T1331 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4] 2> 381957 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 381958 T1331 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4] 2> 381960 T1346 C4809 P57702 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 382010 T1346 C4809 P57702 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> 382010 T1346 C4809 P57702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4] 2> 382011 T1346 C4809 P57702 oass.SolrIndexSearcher.<init> Opening Searcher@791cd4 main
[junit4:junit4] 2> 382011 T1346 C4809 P57702 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 382021 T1357 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@791cd4 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> ASYNC NEW_CORE C4814 name=collection1 org.apache.solr.core.SolrCore@18c7b8d url=http://127.0.0.1:57702/collection1 node=127.0.0.1:57702_ C4814_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:57702_, base_url=http://127.0.0.1:57702, leader=true}
[junit4:junit4] 2> 382022 T1346 C4814 P57702 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 62
[junit4:junit4] 2> ASYNC NEW_CORE C4815 name=collection1 org.apache.solr.core.SolrCore@6fd785 url=http://127.0.0.1:35789/collection1 node=127.0.0.1:35789_ C4815_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:35789_, base_url=http://127.0.0.1:35789}
[junit4:junit4] 2> 382024 T1383 C4815 P35789 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:54846/collection1/, StdNode: http://127.0.0.1:35789/collection1/, StdNode: http://127.0.0.1:41066/collection1/, StdNode: http://127.0.0.1:37613/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4] 2> ASYNC NEW_CORE C4816 name=collection1 org.apache.solr.core.SolrCore@43017e url=http://127.0.0.1:41066/collection1 node=127.0.0.1:41066_ C4816_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:41066_, base_url=http://127.0.0.1:41066}
[junit4:junit4] 2> 382027 T1402 C4816 P41066 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C4817 name=collection1 org.apache.solr.core.SolrCore@1a9d4fc url=http://127.0.0.1:37613/collection1 node=127.0.0.1:37613_ C4817_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:37613_, base_url=http://127.0.0.1:37613}
[junit4:junit4] 2> 382027 T1416 C4817 P37613 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> ASYNC NEW_CORE C4818 name=collection1 org.apache.solr.core.SolrCore@1df5130 url=http://127.0.0.1:54846/collection1 node=127.0.0.1:54846_ C4818_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:54846_, base_url=http://127.0.0.1:54846, leader=true}
[junit4:junit4] 2> 382027 T1367 C4818 P54846 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 382028 T1385 C4815 P35789 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 382050 T1416 C4817 P37613 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_6,generation=6}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_7,generation=7}
[junit4:junit4] 2> 382050 T1416 C4817 P37613 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 7
[junit4:junit4] 2> 382051 T1416 C4817 P37613 oass.SolrIndexSearcher.<init> Opening Searcher@1b6784 main
[junit4:junit4] 2> 382051 T1416 C4817 P37613 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 382052 T1429 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1b6784 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4] 2> 382052 T1416 C4817 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 25
[junit4:junit4] 2> 382060 T1402 C4816 P41066 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_5,generation=5}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_6,generation=6}
[junit4:junit4] 2> 382060 T1402 C4816 P41066 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4] 2> 382060 T1367 C4818 P54846 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_6,generation=6}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_7,generation=7}
[junit4:junit4] 2> 382061 T1367 C4818 P54846 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 7
[junit4:junit4] 2> 382062 T1367 C4818 P54846 oass.SolrIndexSearcher.<init> Opening Searcher@1e7f981 main
[junit4:junit4] 2> 382062 T1367 C4818 P54846 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2
[...truncated too long message...]
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:37613_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:37613"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"127.0.0.1:57702__collection1":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "node_name":"127.0.0.1:57702_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:57702",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (4)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:54846_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:41066_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:57702_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:37613_ (0)
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/collections (2)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89908153904726030-127.0.0.1:54846__collection1-n_0000000004 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89908153904726026-127.0.0.1:41066__collection1-n_0000000002 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89908153904726028-127.0.0.1:37613__collection1-n_0000000003 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (1)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:41066_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:41066"}
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89908153904726019-127.0.0.1:57702__collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:57702_",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:57702"}
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/election (4)
[junit4:junit4] 1> /solr/overseer_elect/election/89908153904726028-127.0.0.1:37613_-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89908153904726019-127.0.0.1:57702_-n_0000000000 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89908153904726030-127.0.0.1:54846_-n_0000000005 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89908153904726026-127.0.0.1:41066_-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89908153904726019-127.0.0.1:57702_-n_0000000000"}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=CB968363466C6444 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_PA -Dtests.timezone=Asia/Tokyo -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE 101s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 305 from http://127.0.0.1:54846/collection1lastClient and got 247 from http://127.0.0.1:41066/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([CB968363466C6444:4A700D7B31330478]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 461253 T1331 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 101277 T1330 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 461310 T1406 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 461311 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 461311 T1406 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=Pulsing41(freqCutoff=6 minBlockSize=66 maxBlockSize=163), text=PostingsFormat(name=Direct), _version_=Pulsing41(freqCutoff=6 minBlockSize=66 maxBlockSize=163), rnd_b=Pulsing41(freqCutoff=6 minBlockSize=66 maxBlockSize=163), intDefault=Pulsing41(freqCutoff=6 minBlockSize=66 maxBlockSize=163), timestamp=Pulsing41(freqCutoff=6 minBlockSize=66 maxBlockSize=163), id=Pulsing41(freqCutoff=6 minBlockSize=66 maxBlockSize=163), a_t=Pulsing41(freqCutoff=6 minBlockSize=66 maxBlockSize=163), range_facet_sl=PostingsFormat(name=Direct), range_facet_si=PostingsFormat(name=SimpleText), other_tl1=Pulsing41(freqCutoff=6 minBlockSize=66 maxBlockSize=163), multiDefault=PostingsFormat(name=SimpleText), a_si=PostingsFormat(name=SimpleText)}, docValues:{timestamp=DocValuesFormat(name=CheapBastard)}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=es_PA, timezone=Asia/Tokyo
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation 1.7.0_25 (32-bit)/cpus=8,threads=1,free=25810176,total=100057088
[junit4:junit4] 2> NOTE: All tests run in this JVM: [DocValuesTest, SpellingQueryConverterTest, TestPerFieldSimilarity, TestPropInjectDefaults, SpatialFilterTest, SynonymTokenizerTest, AlternateDirectoryTest, TestPseudoReturnFields, TestGroupingSearch, TestStressVersions, TestReversedWildcardFilterFactory, ClusterStateUpdateTest, PreAnalyzedFieldTest, TestSolrQueryParser, MBeansHandlerTest, TestSolrIndexConfig, TermsComponentTest, TestDocSet, SuggesterFSTTest, JsonLoaderTest, RecoveryZkTest, DirectSolrConnectionTest, SOLR749Test, TestStressRecovery, ResourceLoaderTest, RegexBoostProcessorTest, FieldMutatingUpdateProcessorTest, TestPhraseSuggestions, TestReplicationHandler, TestPluginEnable, DistributedQueryElevationComponentTest, TestJoin, TestBadConfig, TestUniqueKeyFieldResource, AliasIntegrationTest, TestFastWriter, SliceStateUpdateTest, TestLRUCache, QueryEqualityTest, SyncSliceTest]
[junit4:junit4] Completed on J0 in 101.80s, 1 test, 1 failure <<< FAILURES!
[...truncated 620 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1253 tests, 1 failure, 13 ignored (7 assumptions)
Total time: 40 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_25 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure