You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/08/13 08:24:11 UTC
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b102) - Build
# 6994 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6994/
Java: 64bit/jdk1.8.0-ea-b102 -XX:+UseCompressedOops -XX:+UseParallelGC
1 tests failed.
FAILED: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch
Error Message:
Server at http://127.0.0.1:60445/e_q/tv/collection1 returned non ok status:500, message:Server Error
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:60445/e_q/tv/collection1 returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([7FE854184403A55F:FE0EDA00335CC563]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:387)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexQueryDeleteHierarchical(FullSolrCloudDistribCmdsTest.java:259)
at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:124)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:491)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9626 lines...]
[junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
[junit4] 2> 247575 T1045 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /e_q/tv
[junit4] 2> 247597 T1045 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-FullSolrCloudDistribCmdsTest-1376374225614
[junit4] 2> 247598 T1045 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 247613 T1046 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 247713 T1045 oasc.ZkTestServer.run start zk server on port:41526
[junit4] 2> 247717 T1045 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 247792 T1052 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@278b35b8 name:ZooKeeperConnection Watcher:127.0.0.1:41526 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 247792 T1045 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 247793 T1045 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 247805 T1045 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 247810 T1054 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@294cdf39 name:ZooKeeperConnection Watcher:127.0.0.1:41526/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 247811 T1045 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 247811 T1045 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 247813 T1045 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 247815 T1045 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 247817 T1045 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 247819 T1045 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 247821 T1045 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 247825 T1045 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 247825 T1045 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 247828 T1045 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 247829 T1045 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 247831 T1045 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 247831 T1045 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 247834 T1045 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 247834 T1045 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 247836 T1045 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 247837 T1045 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 247839 T1045 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 247839 T1045 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 247848 T1045 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 247848 T1045 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 247851 T1045 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 247852 T1045 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 247854 T1045 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 247854 T1045 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 247925 T1045 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 247928 T1045 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60445
[junit4] 2> 247929 T1045 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 247929 T1045 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 247929 T1045 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376374225873
[junit4] 2> 247930 T1045 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376374225873/'
[junit4] 2> 247939 T1045 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376374225873/solr.xml
[junit4] 2> 247946 T1045 oasc.CoreContainer.<init> New CoreContainer 1324016878
[junit4] 2> 247946 T1045 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376374225873/]
[junit4] 2> 247947 T1045 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 247947 T1045 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 247947 T1045 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 247948 T1045 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 247948 T1045 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 247948 T1045 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 247948 T1045 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 247949 T1045 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 247949 T1045 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 247949 T1045 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 247954 T1045 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41526/solr
[junit4] 2> 247954 T1045 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 247955 T1045 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 247957 T1065 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48d34020 name:ZooKeeperConnection Watcher:127.0.0.1:41526 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 247957 T1045 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 247960 T1045 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 247969 T1045 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 247973 T1067 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3cc7e167 name:ZooKeeperConnection Watcher:127.0.0.1:41526/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 247974 T1045 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 247975 T1045 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 247978 T1045 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 247981 T1045 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 247982 T1045 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60445_e_q%2Ftv
[junit4] 2> 247983 T1045 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60445_e_q%2Ftv
[junit4] 2> 247985 T1045 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 247990 T1045 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 247992 T1045 oasc.Overseer.start Overseer (id=90202061252132867-127.0.0.1:60445_e_q%2Ftv-n_0000000000) starting
[junit4] 2> 247998 T1045 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 248002 T1068 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 248006 T1045 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 248007 T1069 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 248009 T1045 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 248013 T1045 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 248017 T1070 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 248017 T1070 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 248018 T1070 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 249507 T1068 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 249508 T1068 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:60445_e_q%2Ftv",
[junit4] 2> "base_url":"http://127.0.0.1:60445/e_q/tv"}
[junit4] 2> 249508 T1068 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 249509 T1068 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 249513 T1067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 250018 T1070 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 250018 T1070 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376374225873/collection1
[junit4] 2> 250019 T1070 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 250019 T1070 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 250019 T1070 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 250020 T1070 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376374225873/collection1/'
[junit4] 2> 250021 T1070 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376374225873/collection1/lib/README' to classloader
[junit4] 2> 250021 T1070 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376374225873/collection1/lib/classes/' to classloader
[junit4] 2> 250048 T1070 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 250062 T1070 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 250063 T1070 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 250070 T1070 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 250182 T1070 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 250187 T1070 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 250188 T1070 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 250198 T1070 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 250201 T1070 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 250204 T1070 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 250205 T1070 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 250205 T1070 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 250205 T1070 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 250206 T1070 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 250206 T1070 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 250207 T1070 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 250207 T1070 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376374225873/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/control/data/
[junit4] 2> 250207 T1070 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b1cff57
[junit4] 2> 250208 T1070 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/control/data
[junit4] 2> 250208 T1070 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/control/data/index/
[junit4] 2> 250208 T1070 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 250208 T1070 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/control/data/index
[junit4] 2> 250210 T1070 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6aac842c lockFactory=org.apache.lucene.store.NativeFSLockFactory@32426ae5),segFN=segments_1,generation=1}
[junit4] 2> 250210 T1070 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 250214 T1070 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 250214 T1070 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 250215 T1070 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 250215 T1070 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 250215 T1070 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 250216 T1070 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 250216 T1070 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 250216 T1070 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 250216 T1070 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 250218 T1070 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 250219 T1070 oass.SolrIndexSearcher.<init> Opening Searcher@6ac942c main
[junit4] 2> 250219 T1070 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 250220 T1070 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 250222 T1071 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6ac942c main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 250222 T1070 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 250222 T1070 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60445/e_q/tv collection:control_collection shard:shard1
[junit4] 2> 250222 T1070 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 250226 T1070 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 250234 T1070 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 250236 T1070 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 250236 T1070 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 250236 T1070 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60445/e_q/tv/collection1/
[junit4] 2> 250236 T1070 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 250236 T1070 oasc.SyncStrategy.syncToMe http://127.0.0.1:60445/e_q/tv/collection1/ has no replicas
[junit4] 2> 250236 T1070 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60445/e_q/tv/collection1/ shard1
[junit4] 2> 250236 T1070 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 251016 T1068 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 251021 T1067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 251042 T1070 oasc.ZkController.register We are http://127.0.0.1:60445/e_q/tv/collection1/ and leader is http://127.0.0.1:60445/e_q/tv/collection1/
[junit4] 2> 251042 T1070 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60445/e_q/tv
[junit4] 2> 251042 T1070 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 251042 T1070 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 251042 T1070 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 251044 T1070 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 251045 T1045 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 251045 T1045 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 251046 T1045 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 251049 T1045 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4] 2> 251050 T1045 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 251051 T1074 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ae4a3fc name:ZooKeeperConnection Watcher:127.0.0.1:41526/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 251052 T1045 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 251053 T1045 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 251054 T1045 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4] 2> 251154 T1045 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 251156 T1045 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57410
[junit4] 2> 251162 T1045 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 251162 T1045 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 251163 T1045 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376374229071
[junit4] 2> 251163 T1045 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376374229071/'
[junit4] 2> 251177 T1045 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376374229071/solr.xml
[junit4] 2> 251186 T1045 oasc.CoreContainer.<init> New CoreContainer 1337696710
[junit4] 2> 251186 T1045 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376374229071/]
[junit4] 2> 251187 T1045 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 251187 T1045 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 251187 T1045 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 251188 T1045 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 251188 T1045 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 251188 T1045 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 251189 T1045 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 251189 T1045 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 251189 T1045 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 251190 T1045 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 251194 T1045 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41526/solr
[junit4] 2> 251194 T1045 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 251195 T1045 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 251197 T1085 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a29312 name:ZooKeeperConnection Watcher:127.0.0.1:41526 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 251197 T1045 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 251202 T1045 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 251202 T1047 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x140764bfaf10005, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4] 2>
[junit4] 2> 251206 T1045 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 251207 T1087 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@750135a4 name:ZooKeeperConnection Watcher:127.0.0.1:41526/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 251207 T1045 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 251210 T1045 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 252212 T1045 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57410_e_q%2Ftv
[junit4] 2> 252215 T1045 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57410_e_q%2Ftv
[junit4] 2> 252218 T1074 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 252218 T1067 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 252218 T1067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 252218 T1087 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 252222 T1088 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 252222 T1088 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 252223 T1088 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 252525 T1068 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 252526 T1068 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:60445_e_q%2Ftv",
[junit4] 2> "base_url":"http://127.0.0.1:60445/e_q/tv"}
[junit4] 2> 252529 T1068 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:57410_e_q%2Ftv",
[junit4] 2> "base_url":"http://127.0.0.1:57410/e_q/tv"}
[junit4] 2> 252529 T1068 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 252529 T1068 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 252532 T1087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 252532 T1067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 252532 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 253223 T1088 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 253223 T1088 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376374229071/collection1
[junit4] 2> 253223 T1088 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 253224 T1088 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 253224 T1088 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 253225 T1088 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376374229071/collection1/'
[junit4] 2> 253226 T1088 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376374229071/collection1/lib/README' to classloader
[junit4] 2> 253226 T1088 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376374229071/collection1/lib/classes/' to classloader
[junit4] 2> 253244 T1088 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 253257 T1088 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 253258 T1088 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 253264 T1088 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 253361 T1088 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 253364 T1088 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 253365 T1088 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 253373 T1088 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 253375 T1088 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 253377 T1088 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 253378 T1088 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 253378 T1088 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 253378 T1088 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 253378 T1088 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 253379 T1088 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 253379 T1088 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 253379 T1088 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376374229071/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/jetty1/
[junit4] 2> 253380 T1088 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b1cff57
[junit4] 2> 253380 T1088 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/jetty1
[junit4] 2> 253380 T1088 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/jetty1/index/
[junit4] 2> 253381 T1088 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 253381 T1088 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/jetty1/index
[junit4] 2> 253382 T1088 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@39e5e5f4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@62628ff6),segFN=segments_1,generation=1}
[junit4] 2> 253382 T1088 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 253384 T1088 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 253385 T1088 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 253385 T1088 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 253385 T1088 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 253386 T1088 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 253386 T1088 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 253386 T1088 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 253386 T1088 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 253386 T1088 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 253388 T1088 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 253389 T1088 oass.SolrIndexSearcher.<init> Opening Searcher@290ddfb9 main
[junit4] 2> 253389 T1088 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 253389 T1088 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 253391 T1089 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@290ddfb9 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 253392 T1088 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 253393 T1088 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57410/e_q/tv collection:collection1 shard:shard1
[junit4] 2> 253393 T1088 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 253396 T1088 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 253400 T1088 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 253408 T1088 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 253408 T1088 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 253409 T1088 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57410/e_q/tv/collection1/
[junit4] 2> 253409 T1088 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 253409 T1088 oasc.SyncStrategy.syncToMe http://127.0.0.1:57410/e_q/tv/collection1/ has no replicas
[junit4] 2> 253409 T1088 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57410/e_q/tv/collection1/ shard1
[junit4] 2> 253409 T1088 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 254040 T1068 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 254045 T1067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 254045 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 254045 T1087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 254067 T1088 oasc.ZkController.register We are http://127.0.0.1:57410/e_q/tv/collection1/ and leader is http://127.0.0.1:57410/e_q/tv/collection1/
[junit4] 2> 254067 T1088 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57410/e_q/tv
[junit4] 2> 254067 T1088 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 254067 T1088 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 254067 T1088 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 254068 T1088 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 254069 T1045 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 254070 T1045 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 254070 T1045 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 254163 T1045 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 254164 T1045 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36051
[junit4] 2> 254166 T1045 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 254166 T1045 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 254166 T1045 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376374232089
[junit4] 2> 254166 T1045 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376374232089/'
[junit4] 2> 254175 T1045 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376374232089/solr.xml
[junit4] 2> 254182 T1045 oasc.CoreContainer.<init> New CoreContainer 986480138
[junit4] 2> 254182 T1045 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376374232089/]
[junit4] 2> 254182 T1045 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 254183 T1045 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 254183 T1045 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 254183 T1045 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 254183 T1045 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 254183 T1045 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 254184 T1045 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 254184 T1045 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 254184 T1045 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 254184 T1045 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 254187 T1045 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41526/solr
[junit4] 2> 254187 T1045 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 254188 T1045 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 254189 T1101 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c57bf86 name:ZooKeeperConnection Watcher:127.0.0.1:41526 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 254190 T1045 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 254191 T1045 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 254194 T1045 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 254195 T1103 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54f74a00 name:ZooKeeperConnection Watcher:127.0.0.1:41526/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 254195 T1045 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 254198 T1045 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 255200 T1045 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36051_e_q%2Ftv
[junit4] 2> 255201 T1045 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36051_e_q%2Ftv
[junit4] 2> 255202 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 255203 T1087 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 255203 T1103 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 255203 T1067 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 255204 T1067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 255203 T1087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 255205 T1074 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 255207 T1104 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 255207 T1104 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 255207 T1104 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 255549 T1068 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 255550 T1068 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:57410_e_q%2Ftv",
[junit4] 2> "base_url":"http://127.0.0.1:57410/e_q/tv"}
[junit4] 2> 255555 T1068 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:36051_e_q%2Ftv",
[junit4] 2> "base_url":"http://127.0.0.1:36051/e_q/tv"}
[junit4] 2> 255555 T1068 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 255555 T1068 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 255558 T1067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 255558 T1103 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 255558 T1087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 255558 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 256208 T1104 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 256208 T1104 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376374232089/collection1
[junit4] 2> 256208 T1104 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 256209 T1104 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 256209 T1104 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 256209 T1104 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376374232089/collection1/'
[junit4] 2> 256210 T1104 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376374232089/collection1/lib/README' to classloader
[junit4] 2> 256210 T1104 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376374232089/collection1/lib/classes/' to classloader
[junit4] 2> 256223 T1104 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 256230 T1104 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 256231 T1104 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 256235 T1104 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 256334 T1104 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 256339 T1104 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 256341 T1104 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 256349 T1104 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 256351 T1104 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 256354 T1104 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 256355 T1104 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 256355 T1104 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 256355 T1104 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 256356 T1104 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 256356 T1104 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 256356 T1104 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 256357 T1104 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376374232089/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/jetty2/
[junit4] 2> 256357 T1104 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b1cff57
[junit4] 2> 256358 T1104 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/jetty2
[junit4] 2> 256358 T1104 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/jetty2/index/
[junit4] 2> 256358 T1104 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 256358 T1104 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/jetty2/index
[junit4] 2> 256360 T1104 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@38ae520e lockFactory=org.apache.lucene.store.NativeFSLockFactory@45b2bba9),segFN=segments_1,generation=1}
[junit4] 2> 256360 T1104 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 256363 T1104 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 256363 T1104 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 256364 T1104 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 256365 T1104 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 256365 T1104 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 256365 T1104 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 256365 T1104 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 256366 T1104 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 256366 T1104 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 256368 T1104 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 256369 T1104 oass.SolrIndexSearcher.<init> Opening Searcher@60579a95 main
[junit4] 2> 256370 T1104 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 256370 T1104 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 256373 T1105 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@60579a95 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 256375 T1104 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 256375 T1104 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36051/e_q/tv collection:collection1 shard:shard2
[junit4] 2> 256375 T1104 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 256379 T1104 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 256383 T1104 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 256385 T1104 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 256385 T1104 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 256385 T1104 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:36051/e_q/tv/collection1/
[junit4] 2> 256386 T1104 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 256386 T1104 oasc.SyncStrategy.syncToMe http://127.0.0.1:36051/e_q/tv/collection1/ has no replicas
[junit4] 2> 256386 T1104 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:36051/e_q/tv/collection1/ shard2
[junit4] 2> 256386 T1104 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 257062 T1068 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 257067 T1087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 257067 T1067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 257067 T1103 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 257067 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 257092 T1104 oasc.ZkController.register We are http://127.0.0.1:36051/e_q/tv/collection1/ and leader is http://127.0.0.1:36051/e_q/tv/collection1/
[junit4] 2> 257092 T1104 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36051/e_q/tv
[junit4] 2> 257092 T1104 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 257092 T1104 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 257092 T1104 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 257094 T1104 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 257094 T1045 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 257095 T1045 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 257095 T1045 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 257155 T1045 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 257157 T1045 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37896
[junit4] 2> 257158 T1045 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 257158 T1045 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 257158 T1045 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376374235113
[junit4] 2> 257159 T1045 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376374235113/'
[junit4] 2> 257171 T1045 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376374235113/solr.xml
[junit4] 2> 257177 T1045 oasc.CoreContainer.<init> New CoreContainer 998448983
[junit4] 2> 257178 T1045 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376374235113/]
[junit4] 2> 257178 T1045 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 257178 T1045 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 257178 T1045 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 257179 T1045 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 257179 T1045 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 257179 T1045 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 257179 T1045 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 257179 T1045 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 257180 T1045 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 257180 T1045 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 257182 T1045 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41526/solr
[junit4] 2> 257183 T1045 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 257183 T1045 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 257185 T1117 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5660d0a7 name:ZooKeeperConnection Watcher:127.0.0.1:41526 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 257185 T1045 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 257189 T1045 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 257193 T1045 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 257195 T1119 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15eccd32 name:ZooKeeperConnection Watcher:127.0.0.1:41526/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 257196 T1045 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 257198 T1045 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 258200 T1045 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37896_e_q%2Ftv
[junit4] 2> 258201 T1045 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37896_e_q%2Ftv
[junit4] 2> 258203 T1103 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 258203 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 258204 T1067 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 258204 T1087 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 258204 T1119 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 258204 T1087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 258204 T1067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 258205 T1103 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 258205 T1074 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 258208 T1120 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 258208 T1120 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 258209 T1120 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 258571 T1068 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 258572 T1068 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard2",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:36051_e_q%2Ftv",
[junit4] 2> "base_url":"http://127.0.0.1:36051/e_q/tv"}
[junit4] 2> 258575 T1068 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:37896_e_q%2Ftv",
[junit4] 2> "base_url":"http://127.0.0.1:37896/e_q/tv"}
[junit4] 2> 258575 T1068 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 258575 T1068 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 258578 T1087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 258578 T1103 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 258578 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 258578 T1067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 258578 T1119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 259209 T1120 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 259209 T1120 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376374235113/collection1
[junit4] 2> 259210 T1120 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 259210 T1120 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 259210 T1120 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 259211 T1120 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376374235113/collection1/'
[junit4] 2> 259211 T1120 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376374235113/collection1/lib/README' to classloader
[junit4] 2> 259212 T1120 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376374235113/collection1/lib/classes/' to classloader
[junit4] 2> 259230 T1120 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 259238 T1120 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 259240 T1120 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 259246 T1120 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 259363 T1120 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 259369 T1120 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 259370 T1120 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 259379 T1120 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 259382 T1120 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 259384 T1120 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 259385 T1120 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 259385 T1120 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 259385 T1120 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 259386 T1120 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 259386 T1120 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 259386 T1120 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 259387 T1120 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376374235113/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/jetty3/
[junit4] 2> 259387 T1120 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7b1cff57
[junit4] 2> 259388 T1120 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/jetty3
[junit4] 2> 259388 T1120 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/jetty3/index/
[junit4] 2> 259388 T1120 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 259388 T1120 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376374225613/jetty3/index
[junit4] 2> 259390 T1120 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5df576ae lockFactory=org.apache.lucene.store.NativeFSLockFactory@4db8fc81),segFN=segments_1,generation=1}
[junit4] 2> 259390 T1120 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 259394 T1120 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 259395 T1120 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 259395 T1120 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 259396 T1120 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 259396 T1120 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 259396 T1120 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 259397 T1120 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 259397 T1120 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 259397 T1120 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 259400 T1120 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 259401 T1120 oass.SolrIndexSearcher.<init> Opening Searcher@a128a94 main
[junit4] 2> 259402 T1120 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 259402 T1120 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 259405 T1121 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a128a94 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 259407 T1120 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 259407 T1120 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37896/e_q/tv collection:collection1 shard:shard1
[junit4] 2> 259407 T1120 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 259413 T1120 oasc.ZkController.register We are http://127.0.0.1:37896/e_q/tv/collection1/ and leader is http://127.0.0.1:57410/e_q/tv/collection1/
[junit4] 2> 259413 T1120 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37896/e_q/tv
[junit4] 2> 259414 T1120 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 259414 T1120 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C281 name=collection1 org.apache.solr.core.SolrCore@62a00558 url=http://127.0.0.1:37896/e_q/tv/collection1 node=127.0.0.1:37896_e_q%2Ftv C281_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:37896_e_q%2Ftv, base_url=http://127.0.0.1:37896/e_q/tv}
[junit4] 2> 259414 T1122 C281 P37896 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 259415 T1120 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 259416 T1045 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 259417 T1045 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 259417 T1122 C281 P37896 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 259417 T1045 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 259417 T1122 C281 P37896 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4] 2> 259417 T1122 C281 P37896 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 259420 T1122 C281 P37896 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 259424 T1082 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 259497 T1045 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 259499 T1045 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49374
[junit4] 2> 259500 T1045 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 259500 T1045 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 259501 T1045 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376374237436
[junit4] 2> 259501 T1045 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376374237436/'
[junit4] 2> 259510 T1045 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376374237436/solr.xml
[junit4] 2> 259516 T1045 oasc.CoreContainer.<init> New CoreContainer 2147037807
[junit4] 2> 259516 T1045 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376374237436/]
[junit4] 2> 259516 T1045 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 259517 T1045 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 259517 T1045 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 259517 T1045 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 259517 T1045 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 259518 T1045 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 259518 T1045 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 259518 T1045 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 259518 T1045 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 259518 T1045 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 259521 T1045 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41526/solr
[junit4] 2> 259521 T1045 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 259522 T1045 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 259523 T1134 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6df7adc2 name:ZooKeeperConnection Watcher:127.0.0.1:41526 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 259524 T1045 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 259525 T1045 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 259528 T1045 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 259529 T1136 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@573aa22d name:ZooKeeperConnection Watcher:127.0.0.1:41526/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 259530 T1045 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 259532 T1045 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 260082 T1068 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 260083 T1068 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"recovering",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:37896_e_q%2Ftv",
[junit4] 2> "base_url":"http://127.0.0.1:37896/e_q/tv"}
[junit4] 2> 260088 T1136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 260088 T1103 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 260088 T1087 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 260089 T1119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 260089 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 260088 T1067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 260425 T1082 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 260425 T1082 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node3&state=recovering&nodeName=127.0.0.1:37896_e_q%252Ftv&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4] 2> 260534 T1045 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49374_e_q%2Ftv
[junit4] 2> 260535 T1045 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49374_e_q%2Ftv
[junit4] 2> 260537 T1103 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 260537 T1074 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 260537 T1119 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 260537 T1136 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 260538 T1087 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 260538 T1087 oascc.ZkStateReader$2.proces
[...truncated too long message...]
9grand32_s=PostingsFormat(name=MockRandom), 65grand58_s=PostingsFormat(name=Memory doPackFST= true), 35grand28_s=PostingsFormat(name=Memory doPackFST= true), 68grand13_s=PostingsFormat(name=Memory doPackFST= true), 3child1_s=PostingsFormat(name=Memory doPackFST= true), 8grand0_s=PostingsFormat(name=Memory doPackFST= true), 55grand1_s=PostingsFormat(name=Asserting), 49grand24_s=PostingsFormat(name=Asserting), 36child31_s=PostingsFormat(name=MockRandom), 41grand8_s=PostingsFormat(name=Asserting), 27grand25_s=PostingsFormat(name=NestedPulsing), 29child1_s=PostingsFormat(name=Memory doPackFST= true), 50grand43_s=PostingsFormat(name=Memory doPackFST= true), 53grand3_s=PostingsFormat(name=Asserting), 46grand22_s=PostingsFormat(name=NestedPulsing), 48grand40_s=PostingsFormat(name=Memory doPackFST= true), 55grand29_s=PostingsFormat(name=MockRandom), 63grand7_s=PostingsFormat(name=Memory doPackFST= true), 30grand0_s=PostingsFormat(name=Asserting), 25child10_s=PostingsFormat(name=Memory doPackFST= true), 39grand21_s=PostingsFormat(name=MockRandom), 6child5_s=PostingsFormat(name=Asserting), 46grand33_s=PostingsFormat(name=NestedPulsing), 65grand25_s=PostingsFormat(name=Memory doPackFST= true), 41grand31_s=PostingsFormat(name=MockRandom), 6grand4_s=PostingsFormat(name=NestedPulsing), 60grand54_s=PostingsFormat(name=MockRandom), 52grand26_s=PostingsFormat(name=MockRandom), 59grand25_s=PostingsFormat(name=Asserting), 59grand53_s=PostingsFormat(name=NestedPulsing), 55grand42_s=PostingsFormat(name=NestedPulsing), 37grand4_s=PostingsFormat(name=Memory doPackFST= true), 71grand0_s=PostingsFormat(name=NestedPulsing), 57grand43_s=PostingsFormat(name=Asserting), 39grand7_s=PostingsFormat(name=MockRandom), 13child2_s=PostingsFormat(name=Memory doPackFST= true), 70grand58_s=PostingsFormat(name=NestedPulsing), 14grand9_s=PostingsFormat(name=Memory doPackFST= true), 61grand3_s=PostingsFormat(name=NestedPulsing), 50grand18_s=PostingsFormat(name=Memory doPackFST= true), 67grand15_s=PostingsFormat(name=Asserting), 71grand25_s=PostingsFormat(name=Asserting), 55grand21_s=PostingsFormat(name=MockRandom), 22child12_s=PostingsFormat(name=MockRandom), 43grand17_s=PostingsFormat(name=Asserting), 30grand29_s=PostingsFormat(name=NestedPulsing), 58grand50_s=PostingsFormat(name=Memory doPackFST= true), 53grand27_s=PostingsFormat(name=MockRandom), 17grand15_s=PostingsFormat(name=NestedPulsing), 33grand6_s=PostingsFormat(name=NestedPulsing), 32grand16_s=PostingsFormat(name=NestedPulsing), 24grand9_s=PostingsFormat(name=Asserting), 53grand38_s=PostingsFormat(name=MockRandom), 0parent_f1_s=PostingsFormat(name=Memory doPackFST= true), 34grand26_s=PostingsFormat(name=Asserting), 68grand40_s=PostingsFormat(name=NestedPulsing), 59grand0_s=PostingsFormat(name=Memory doPackFST= true), 63grand24_s=PostingsFormat(name=MockRandom), 52grand18_s=PostingsFormat(name=NestedPulsing), 64grand7_s=PostingsFormat(name=MockRandom), 9grand2_s=PostingsFormat(name=MockRandom), 20grand5_s=PostingsFormat(name=Asserting), 23child6_s=PostingsFormat(name=Asserting), 29child18_s=PostingsFormat(name=Memory doPackFST= true), 40grand18_s=PostingsFormat(name=Asserting), 71grand19_s=PostingsFormat(name=NestedPulsing), 65grand36_s=PostingsFormat(name=Memory doPackFST= true), 34child16_s=PostingsFormat(name=Memory doPackFST= true), 64grand48_s=PostingsFormat(name=NestedPulsing), 26child13_s=PostingsFormat(name=NestedPulsing), 35grand34_s=PostingsFormat(name=Asserting), 65grand42_s=PostingsFormat(name=Asserting), 64grand53_s=PostingsFormat(name=Memory doPackFST= true), 39grand10_s=PostingsFormat(name=MockRandom), 18grand16_s=PostingsFormat(name=NestedPulsing), 34grand20_s=PostingsFormat(name=MockRandom), 69grand44_s=PostingsFormat(name=MockRandom), 5grand0_s=PostingsFormat(name=MockRandom), 71grand14_s=PostingsFormat(name=Asserting), 7grand2_s=PostingsFormat(name=MockRandom), 65grand7_s=PostingsFormat(name=NestedPulsing), 37grand23_s=PostingsFormat(name=MockRandom), 57grand54_s=PostingsFormat(name=Asserting), 31grand3_s=PostingsFormat(name=Asserting), 20child5_s=PostingsFormat(name=Asserting), 59grand36_s=PostingsFormat(name=Asserting), 35child9_s=PostingsFormat(name=Asserting), 34grand15_s=PostingsFormat(name=Asserting), 28child7_s=PostingsFormat(name=MockRandom), 33child32_s=PostingsFormat(name=MockRandom), 62grand1_s=PostingsFormat(name=MockRandom), 70grand6_s=PostingsFormat(name=MockRandom), 45grand21_s=PostingsFormat(name=NestedPulsing), 58grand54_s=PostingsFormat(name=Memory doPackFST= true), 16grand11_s=PostingsFormat(name=Asserting), 29child24_s=PostingsFormat(name=MockRandom), 36grand14_s=PostingsFormat(name=Memory doPackFST= true), 41grand22_s=PostingsFormat(name=Asserting), 56grand22_s=PostingsFormat(name=MockRandom), 18child14_s=PostingsFormat(name=Memory doPackFST= true), 49grand30_s=PostingsFormat(name=Memory doPackFST= true), 52grand7_s=PostingsFormat(name=Memory doPackFST= true), 62grand55_s=PostingsFormat(name=Memory doPackFST= true), 38grand13_s=PostingsFormat(name=MockRandom), 65grand20_s=PostingsFormat(name=Asserting), 14child5_s=PostingsFormat(name=Memory doPackFST= true), 28grand20_s=PostingsFormat(name=Memory doPackFST= true), 39grand18_s=PostingsFormat(name=MockRandom), 26child20_s=PostingsFormat(name=NestedPulsing), 59grand41_s=PostingsFormat(name=MockRandom), 31child23_s=PostingsFormat(name=MockRandom), 22grand14_s=PostingsFormat(name=Asserting), 54grand1_s=PostingsFormat(name=Memory doPackFST= true), 20grand12_s=PostingsFormat(name=Asserting), 15child5_s=PostingsFormat(name=MockRandom), 33grand14_s=PostingsFormat(name=MockRandom), 33child0_s=PostingsFormat(name=Memory doPackFST= true), 40grand22_s=PostingsFormat(name=Memory doPackFST= true), 31grand29_s=PostingsFormat(name=Asserting), 45grand5_s=PostingsFormat(name=NestedPulsing), 24child19_s=PostingsFormat(name=NestedPulsing), 58grand7_s=PostingsFormat(name=NestedPulsing), 55grand36_s=PostingsFormat(name=MockRandom), 68grand35_s=PostingsFormat(name=Memory doPackFST= true), 9child0_s=PostingsFormat(name=Asserting), 52grand51_s=PostingsFormat(name=MockRandom), 61grand44_s=PostingsFormat(name=MockRandom), 68grand62_s=PostingsFormat(name=NestedPulsing), 58grand24_s=PostingsFormat(name=Asserting), 58grand35_s=PostingsFormat(name=Asserting), 19child17_s=PostingsFormat(name=NestedPulsing), 70grand64_s=PostingsFormat(name=MockRandom), 33child8_s=PostingsFormat(name=Memory doPackFST= true), 36child27_s=PostingsFormat(name=NestedPulsing), 49grand39_s=PostingsFormat(name=Asserting), 28grand14_s=PostingsFormat(name=Asserting), 57grand33_s=PostingsFormat(name=NestedPulsing), 12child7_s=PostingsFormat(name=Memory doPackFST= true), 51grand50_s=PostingsFormat(name=MockRandom), 32child3_s=PostingsFormat(name=NestedPulsing), 35child34_s=PostingsFormat(name=MockRandom), 40grand16_s=PostingsFormat(name=MockRandom), 31grand19_s=PostingsFormat(name=NestedPulsing), 51grand0_s=PostingsFormat(name=Memory doPackFST= true), 18child7_s=PostingsFormat(name=NestedPulsing), 28child20_s=PostingsFormat(name=Memory doPackFST= true), 14child12_s=PostingsFormat(name=NestedPulsing), 65grand50_s=PostingsFormat(name=Memory doPackFST= true)}, docValues:{timestamp=DocValuesFormat(name=SimpleText)}, sim=DefaultSimilarity, locale=en_GB, timezone=Africa/Asmera
[junit4] 2> NOTE: Linux 3.2.0-51-generic amd64/Oracle Corporation 1.8.0-ea (64-bit)/cpus=8,threads=1,free=79119368,total=457703424
[junit4] 2> NOTE: All tests run in this JVM: [TestIndexSearcher, PrimUtilsTest, TestSolrCoreProperties, TestIndexingPerformance, TestCloudManagedSchema, IndexBasedSpellCheckerTest, SpellCheckComponentTest, TestPluginEnable, TestSchemaVersionResource, BJQParserTest, SliceStateTest, CustomCollectionTest, TestAddFieldRealTimeGet, TestAtomicUpdateErrorCases, RequiredFieldsTest, TestRandomFaceting, PluginInfoTest, ZkSolrClientTest, NoCacheHeaderTest, XmlUpdateRequestHandlerTest, TestJmxIntegration, TestIBSimilarityFactory, HdfsBasicDistributedZk2Test, PingRequestHandlerTest, OutputWriterTest, PreAnalyzedUpdateProcessorTest, TestBinaryField, TestSolr4Spatial, HdfsCollectionsAPIDistributedZkTest, TestZkChroot, SolrCoreCheckLockOnStartupTest, SimplePostToolTest, StandardRequestHandlerTest, TestReloadAndDeleteDocs, TestRangeQuery, DirectSolrSpellCheckerTest, NumericFieldsTest, SolrRequestParserTest, AssignTest, CoreContainerCoreInitFailuresTest, ClusterStateTest, ChaosMonkeySafeLeaderTest, TestCSVLoader, TestSchemaResource, IndexSchemaRuntimeFieldTest, HdfsLockFactoryTest, TestBinaryResponseWriter, TestUtils, BasicDistributedZk2Test, TestCopyFieldCollectionResource, HighlighterConfigTest, TestRecovery, SignatureUpdateProcessorFactoryTest, FullSolrCloudDistribCmdsTest]
[junit4] Completed on J1 in 140.51s, 1 test, 1 error <<< FAILURES!
[...truncated 733 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:389: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:369: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:450: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1230: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:873: There were test failures: 320 suites, 1360 tests, 1 error, 39 ignored (6 assumptions)
Total time: 37 minutes 16 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-ea-b102 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/ibm-j9-jdk7) - Build #
6999 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6999/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
24 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1008, name=HashSessionScavenger-16, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1008, name=HashSessionScavenger-16, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.<init>(Throwable.java:67)
at java.lang.Throwable.<init>(Throwable.java:78)
at java.lang.Error.<init>(Error.java:81)
at java.lang.VirtualMachineError.<init>(VirtualMachineError.java:63)
at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:69)
at java.security.AccessController.getProtectionDomains(Native Method)
at java.security.AccessController.checkPermission(AccessController.java:88)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:560)
at java.lang.Thread.setContextClassLoader(Thread.java:797)
at org.eclipse.jetty.server.session.HashSessionManager.scavenge(HashSessionManager.java:308)
at org.eclipse.jetty.server.session.HashSessionManager$2.run(HashSessionManager.java:284)
at java.util.Timer$TimerImpl.run(Timer.java:296)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=994, name=qtp220461705-994, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=994, name=qtp220461705-994, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
at java.util.HashMap$1.iterator(HashMap.java:672)
at java.util.HashSet.iterator(HashSet.java:173)
at sun.nio.ch.Util$2.iterator(Util.java:315)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:780)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=966, name=SyncThread:0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=966, name=SyncThread:0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:165)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:576)
at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:151)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1026, name=qtp-1726714643-1026, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1026, name=qtp-1726714643-1026, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
at java.util.HashMap$1.iterator(HashMap.java:672)
at java.util.HashSet.iterator(HashSet.java:173)
at sun.nio.ch.Util$2.iterator(Util.java:315)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:780)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=992, name=HashSessionScavenger-15, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=992, name=HashSessionScavenger-15, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.<init>(Throwable.java:67)
at java.lang.Throwable.<init>(Throwable.java:78)
at java.lang.Error.<init>(Error.java:81)
at java.lang.VirtualMachineError.<init>(VirtualMachineError.java:63)
at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:69)
at java.security.AccessController.toArrayOfProtectionDomains(AccessController.java:184)
at java.security.AccessController.checkPermission(AccessController.java:102)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:560)
at java.lang.Thread.setContextClassLoader(Thread.java:797)
at org.eclipse.jetty.server.session.HashSessionManager.scavenge(HashSessionManager.java:308)
at org.eclipse.jetty.server.session.HashSessionManager$2.run(HashSessionManager.java:284)
at java.util.Timer$TimerImpl.run(Timer.java:296)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=993, name=qtp220461705-993, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=993, name=qtp220461705-993, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.<init>(Throwable.java:67)
at java.lang.Throwable.<init>(Throwable.java:78)
at java.lang.Error.<init>(Error.java:81)
at java.lang.VirtualMachineError.<init>(VirtualMachineError.java:63)
at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:69)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:708)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:780)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1029, name=qtp-1726714643-1029, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1029, name=qtp-1726714643-1029, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
at java.util.concurrent.ConcurrentHashMap$KeySet.iterator(ConcurrentHashMap.java:1273)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1.run(SelectorManager.java:712)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:780)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1042, name=qtp1556593299-1042, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1042, name=qtp1556593299-1042, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.addWaiter(AbstractQueuedSynchronizer.java:617)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1208)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:225)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:301)
at org.eclipse.jetty.util.BlockingArrayQueue.offer(BlockingArrayQueue.java:224)
at org.eclipse.jetty.util.thread.QueuedThreadPool.dispatch(QueuedThreadPool.java:359)
at org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.dispatch(SelectChannelConnector.java:300)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:708)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:780)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1048, name=qtp1556593299-1048, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1048, name=qtp1556593299-1048, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1868)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2084)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:780)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1210, name=qtp220461705-1210, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1210, name=qtp220461705-1210, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1868)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2084)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:780)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1109, name=qtp1556593299-1109, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1109, name=qtp1556593299-1109, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
at java.util.concurrent.ConcurrentHashMap$KeySet.iterator(ConcurrentHashMap.java:1273)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1.run(SelectorManager.java:712)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:780)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1106, name=qtp220461705-1106, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1106, name=qtp220461705-1106, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1868)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2084)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:780)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1043, name=qtp1556593299-1043, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1043, name=qtp1556593299-1043, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.<init>(Throwable.java:67)
at java.lang.Throwable.<init>(Throwable.java:78)
at java.lang.Error.<init>(Error.java:81)
at java.lang.VirtualMachineError.<init>(VirtualMachineError.java:63)
at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:69)
at java.util.HashMap$1.iterator(HashMap.java:672)
at java.util.HashSet.iterator(HashSet.java:173)
at sun.nio.ch.Util$2.iterator(Util.java:315)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:780)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1024, name=HashSessionScavenger-17, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1024, name=HashSessionScavenger-17, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
at java.security.AccessController.toArrayOfProtectionDomains(AccessController.java:184)
at java.security.AccessController.checkPermission(AccessController.java:102)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:560)
at java.lang.Thread.setContextClassLoader(Thread.java:797)
at org.eclipse.jetty.server.session.HashSessionManager.scavenge(HashSessionManager.java:344)
at org.eclipse.jetty.server.session.HashSessionManager$2.run(HashSessionManager.java:284)
at java.util.Timer$TimerImpl.run(Timer.java:296)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=964, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=964, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
at java.util.ArrayList$SubList.listIterator(ArrayList.java:1024)
at java.util.AbstractList.listIterator(AbstractList.java:629)
at java.util.ArrayList$SubList.iterator(ArrayList.java:1016)
at java.util.AbstractCollection.toArray(AbstractCollection.java:363)
at java.util.regex.Pattern.split(Pattern.java:1238)
at java.util.regex.Pattern.split(Pattern.java:1270)
at java.lang.String.split(String.java:1749)
at org.apache.zookeeper.jmx.MBeanRegistry.tokenize(MBeanRegistry.java:182)
at org.apache.zookeeper.jmx.MBeanRegistry.makeObjectName(MBeanRegistry.java:204)
at org.apache.zookeeper.jmx.MBeanRegistry.unregister(MBeanRegistry.java:115)
at org.apache.zookeeper.jmx.MBeanRegistry.unregister(MBeanRegistry.java:132)
at org.apache.zookeeper.server.ServerCnxnFactory.unregisterConnection(ServerCnxnFactory.java:137)
at org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:972)
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:352)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:780)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1000, name=qtp220461705-1000, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1000, name=qtp220461705-1000, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
at java.util.concurrent.ConcurrentHashMap$KeySet.iterator(ConcurrentHashMap.java:1273)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1.run(SelectorManager.java:712)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:780)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1096, name=qtp1639013060-1096, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1096, name=qtp1639013060-1096, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1868)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2084)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:780)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1009, name=qtp1639013060-1009 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1009, name=qtp1639013060-1009 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:304)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:780)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1025, name=qtp-1726714643-1025, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1025, name=qtp-1726714643-1025, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
at java.util.HashMap$1.iterator(HashMap.java:672)
at java.util.HashSet.iterator(HashSet.java:173)
at sun.nio.ch.Util$2.iterator(Util.java:315)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:780)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1209, name=qtp1556593299-1209, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1209, name=qtp1556593299-1209, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1868)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2084)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:780)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1041, name=HashSessionScavenger-18, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1041, name=HashSessionScavenger-18, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
at java.security.AccessController.getProtectionDomains(Native Method)
at java.security.AccessController.checkPermission(AccessController.java:88)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:560)
at java.lang.Thread.setContextClassLoader(Thread.java:797)
at org.eclipse.jetty.server.session.HashSessionManager.scavenge(HashSessionManager.java:344)
at org.eclipse.jetty.server.session.HashSessionManager$2.run(HashSessionManager.java:284)
at java.util.Timer$TimerImpl.run(Timer.java:296)
FAILED: org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch
Error Message:
Server at http://127.0.0.1:38771/n_b/m/onenodecollectioncore returned non ok status:404, message:Can not find: /n_b/m/onenodecollectioncore/update
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:38771/n_b/m/onenodecollectioncore returned non ok status:404, message:Can not find: /n_b/m/onenodecollectioncore/update
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976:88DA56E28154494A]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:196)
at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:88)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:613)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:780)
FAILED: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
Build Log:
[...truncated 9663 lines...]
[junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4] 2> 1279686 T3618 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /n_b/m
[junit4] 2> 1279690 T3618 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1376429262138
[junit4] 2> 1279691 T3618 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1279693 T3619 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1279793 T3618 oasc.ZkTestServer.run start zk server on port:56032
[junit4] 2> 1279795 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1280075 T3625 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3afff252 name:ZooKeeperConnection Watcher:127.0.0.1:56032 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1280075 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1280076 T3618 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1280083 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1280084 T3627 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ef341f0e name:ZooKeeperConnection Watcher:127.0.0.1:56032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1280085 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1280085 T3618 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1280088 T3618 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1280090 T3618 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1280092 T3618 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1280094 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1280095 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1280098 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 1280099 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1280101 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1280102 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1280104 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1280105 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1280107 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1280108 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1280110 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1280110 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1280112 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1280113 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1280115 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1280115 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1280118 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1280118 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1280120 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1280121 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1280251 T3618 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1280293 T3618 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56324
[junit4] 2> 1280294 T3618 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1280297 T3618 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1280298 T3618 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571
[junit4] 2> 1280299 T3618 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571/'
[junit4] 2> 1280317 T3618 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571/solr.xml
[junit4] 2> 1280360 T3618 oasc.CoreContainer.<init> New CoreContainer 2074598383
[junit4] 2> 1280361 T3618 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571/]
[junit4] 2> 1280363 T3618 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1280364 T3618 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1280365 T3618 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1280366 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1280367 T3618 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1280368 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1280368 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1280369 T3618 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1280370 T3618 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1280371 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1280382 T3618 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56032/solr
[junit4] 2> 1280383 T3618 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1280385 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1280387 T3638 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72ebed10 name:ZooKeeperConnection Watcher:127.0.0.1:56032 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1280387 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1280389 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1280397 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1280399 T3640 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76ca6e5b name:ZooKeeperConnection Watcher:127.0.0.1:56032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1280399 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1280401 T3618 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1280404 T3618 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1280408 T3618 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1280410 T3618 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56324_n_b%2Fm
[junit4] 2> 1280411 T3618 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56324_n_b%2Fm
[junit4] 2> 1280414 T3618 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1280418 T3618 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1280419 T3618 oasc.Overseer.start Overseer (id=90205668124721155-127.0.0.1:56324_n_b%2Fm-n_0000000000) starting
[junit4] 2> 1280422 T3618 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1280425 T3642 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1280426 T3618 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1280429 T3618 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1280431 T3618 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1280434 T3641 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1280437 T3643 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1280437 T3643 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1280438 T3643 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1281944 T3641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1281945 T3641 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "shard_state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:56324/n_b/m",
[junit4] 2> "state":"down",
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "node_name":"127.0.0.1:56324_n_b%2Fm",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard_range":null,
[junit4] 2> "roles":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard":null,
[junit4] 2> "core":"collection1"}
[junit4] 2> 1281946 T3641 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 1281946 T3641 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1281952 T3640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1282442 T3643 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1282442 T3643 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571/collection1
[junit4] 2> 1282443 T3643 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1282444 T3643 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1282444 T3643 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1282445 T3643 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571/collection1/'
[junit4] 2> 1282447 T3643 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571/collection1/lib/README' to classloader
[junit4] 2> 1282447 T3643 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571/collection1/lib/classes/' to classloader
[junit4] 2> 1282505 T3643 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1282556 T3643 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1282558 T3643 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1282571 T3643 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1283040 T3643 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1283046 T3643 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1283050 T3643 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1283103 T3643 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1283108 T3643 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1283114 T3643 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1283118 T3643 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1283119 T3643 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1283119 T3643 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1283129 T3643 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1283129 T3643 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1283130 T3643 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1283131 T3643 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/control/data/
[junit4] 2> 1283131 T3643 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2f99c27e
[junit4] 2> 1283132 T3643 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/control/data
[junit4] 2> 1283133 T3643 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/control/data/index/
[junit4] 2> 1283134 T3643 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 1283137 T3643 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/control/data/index
[junit4] 2> 1283139 T3643 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@bad203ce lockFactory=org.apache.lucene.store.NativeFSLockFactory@e554df6d),segFN=segments_1,generation=1}
[junit4] 2> 1283140 T3643 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1283144 T3643 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1283145 T3643 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1283145 T3643 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1283147 T3643 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1283147 T3643 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1283149 T3643 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1283149 T3643 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1283166 T3643 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1283167 T3643 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1283170 T3643 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1283186 T3643 oass.SolrIndexSearcher.<init> Opening Searcher@5bf3ca55 main
[junit4] 2> 1283189 T3643 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1283190 T3643 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1283193 T3644 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5bf3ca55 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1283194 T3643 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1283195 T3643 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56324/n_b/m collection:control_collection shard:shard1
[junit4] 2> 1283195 T3643 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 1283204 T3643 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 1283209 T3643 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1283212 T3643 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1283212 T3643 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1283212 T3643 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56324/n_b/m/collection1/
[junit4] 2> 1283213 T3643 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1283213 T3643 oasc.SyncStrategy.syncToMe http://127.0.0.1:56324/n_b/m/collection1/ has no replicas
[junit4] 2> 1283214 T3643 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56324/n_b/m/collection1/ shard1
[junit4] 2> 1283214 T3643 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 1283456 T3641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1283465 T3640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1283520 T3643 oasc.ZkController.register We are http://127.0.0.1:56324/n_b/m/collection1/ and leader is http://127.0.0.1:56324/n_b/m/collection1/
[junit4] 2> 1283520 T3643 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56324/n_b/m
[junit4] 2> 1283521 T3643 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1283521 T3643 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1283522 T3643 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1283523 T3643 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1283524 T3618 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 1283525 T3618 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1283526 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1283533 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4] 2> 1283537 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1283538 T3647 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55c058e0 name:ZooKeeperConnection Watcher:127.0.0.1:56032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1283538 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1283540 T3618 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1283542 T3618 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4] 2> 1283648 T3618 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1283665 T3618 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59579
[junit4] 2> 1283666 T3618 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1283667 T3618 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1283668 T3618 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990
[junit4] 2> 1283668 T3618 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990/'
[junit4] 2> 1283677 T3618 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990/solr.xml
[junit4] 2> 1283746 T3618 oasc.CoreContainer.<init> New CoreContainer 712316351
[junit4] 2> 1283747 T3618 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990/]
[junit4] 2> 1283749 T3618 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1283750 T3618 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1283751 T3618 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1283752 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1283752 T3618 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1283753 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1283754 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1283755 T3618 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1283756 T3618 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1283757 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1283765 T3618 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56032/solr
[junit4] 2> 1283766 T3618 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1283769 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1283770 T3658 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e737e00 name:ZooKeeperConnection Watcher:127.0.0.1:56032 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1283771 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1283782 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1283792 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1283816 T3660 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ae959547 name:ZooKeeperConnection Watcher:127.0.0.1:56032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1283817 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1283821 T3618 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1284824 T3618 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59579_n_b%2Fm
[junit4] 2> 1284827 T3618 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59579_n_b%2Fm
[junit4] 2> 1284830 T3640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1284830 T3647 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1284831 T3660 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1284833 T3640 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1284842 T3661 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1284842 T3661 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1284846 T3661 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1284988 T3641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1284989 T3641 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "shard_state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:56324/n_b/m",
[junit4] 2> "state":"active",
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "node_name":"127.0.0.1:56324_n_b%2Fm",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard_range":null,
[junit4] 2> "roles":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1"}
[junit4] 2> 1284995 T3641 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "shard_state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:59579/n_b/m",
[junit4] 2> "state":"down",
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "node_name":"127.0.0.1:59579_n_b%2Fm",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard_range":null,
[junit4] 2> "roles":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":null,
[junit4] 2> "core":"collection1"}
[junit4] 2> 1284996 T3641 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 1284996 T3641 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1285000 T3647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1285000 T3660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1285000 T3640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1285846 T3661 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1285847 T3661 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990/collection1
[junit4] 2> 1285848 T3661 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1285849 T3661 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1285849 T3661 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1285850 T3661 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990/collection1/'
[junit4] 2> 1285852 T3661 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990/collection1/lib/README' to classloader
[junit4] 2> 1285852 T3661 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990/collection1/lib/classes/' to classloader
[junit4] 2> 1285893 T3661 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1285941 T3661 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1285945 T3661 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1285954 T3661 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1286433 T3661 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1286439 T3661 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1286442 T3661 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1286453 T3661 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1286457 T3661 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1286461 T3661 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1286463 T3661 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1286463 T3661 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1286463 T3661 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1286464 T3661 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1286465 T3661 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1286465 T3661 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1286465 T3661 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty1/
[junit4] 2> 1286466 T3661 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2f99c27e
[junit4] 2> 1286467 T3661 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty1
[junit4] 2> 1286467 T3661 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty1/index/
[junit4] 2> 1286467 T3661 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 1286468 T3661 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty1/index
[junit4] 2> 1286469 T3661 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@ebdce726 lockFactory=org.apache.lucene.store.NativeFSLockFactory@56c9cf76),segFN=segments_1,generation=1}
[junit4] 2> 1286470 T3661 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1286472 T3661 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1286472 T3661 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1286475 T3661 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1286476 T3661 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1286477 T3661 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1286478 T3661 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1286478 T3661 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1286479 T3661 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1286480 T3661 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1286482 T3661 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1286486 T3661 oass.SolrIndexSearcher.<init> Opening Searcher@13a1840f main
[junit4] 2> 1286489 T3661 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1286489 T3661 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1286499 T3661 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1286499 T3661 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59579/n_b/m collection:collection1 shard:shard1
[junit4] 2> 1286500 T3661 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 1286502 T3662 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13a1840f main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1286522 T3661 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1286528 T3661 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1286531 T3661 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1286532 T3661 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1286532 T3661 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59579/n_b/m/collection1/
[junit4] 2> 1286533 T3661 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1286533 T3661 oasc.SyncStrategy.syncToMe http://127.0.0.1:59579/n_b/m/collection1/ has no replicas
[junit4] 2> 1286534 T3661 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59579/n_b/m/collection1/ shard1
[junit4] 2> 1286534 T3661 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1288013 T3641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1288025 T3660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1288025 T3647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1288029 T3640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1288056 T3661 oasc.ZkController.register We are http://127.0.0.1:59579/n_b/m/collection1/ and leader is http://127.0.0.1:59579/n_b/m/collection1/
[junit4] 2> 1288056 T3661 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59579/n_b/m
[junit4] 2> 1288057 T3661 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1288057 T3661 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1288057 T3661 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1288059 T3661 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1288060 T3618 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 1288061 T3618 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1288061 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1288159 T3618 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1288166 T3618 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38771
[junit4] 2> 1288167 T3618 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1288168 T3618 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1288169 T3618 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515
[junit4] 2> 1288170 T3618 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515/'
[junit4] 2> 1288195 T3618 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515/solr.xml
[junit4] 2> 1288243 T3618 oasc.CoreContainer.<init> New CoreContainer -648773389
[junit4] 2> 1288244 T3618 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515/]
[junit4] 2> 1288246 T3618 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1288247 T3618 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1288248 T3618 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1288248 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1288249 T3618 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1288250 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1288252 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1288253 T3618 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1288254 T3618 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1288255 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1288274 T3618 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56032/solr
[junit4] 2> 1288275 T3618 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1288281 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1288283 T3678 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@90e2423d name:ZooKeeperConnection Watcher:127.0.0.1:56032 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1288283 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1288284 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1288291 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1288300 T3680 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@758e0c99 name:ZooKeeperConnection Watcher:127.0.0.1:56032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1288301 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1288306 T3618 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1289310 T3618 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38771_n_b%2Fm
[junit4] 2> 1289312 T3618 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38771_n_b%2Fm
[junit4] 2> 1289315 T3640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1289315 T3647 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1289316 T3680 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1289316 T3647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1289316 T3660 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1289317 T3660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1289317 T3640 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1289342 T3681 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1289342 T3681 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1289344 T3681 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1289539 T3641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1289540 T3641 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "shard_state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:59579/n_b/m",
[junit4] 2> "state":"active",
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "node_name":"127.0.0.1:59579_n_b%2Fm",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard_range":null,
[junit4] 2> "roles":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1"}
[junit4] 2> 1289545 T3641 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "shard_state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:38771/n_b/m",
[junit4] 2> "state":"down",
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "node_name":"127.0.0.1:38771_n_b%2Fm",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard_range":null,
[junit4] 2> "roles":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":null,
[junit4] 2> "core":"collection1"}
[junit4] 2> 1289546 T3641 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1289546 T3641 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1289550 T3660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1289550 T3680 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1289550 T3647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1289550 T3640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1290344 T3681 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1290345 T3681 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515/collection1
[junit4] 2> 1290345 T3681 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1290346 T3681 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1290347 T3681 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1290348 T3681 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515/collection1/'
[junit4] 2> 1290349 T3681 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515/collection1/lib/README' to classloader
[junit4] 2> 1290350 T3681 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515/collection1/lib/classes/' to classloader
[junit4] 2> 1290399 T3681 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1290449 T3681 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1290451 T3681 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1290460 T3681 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1290959 T3681 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1290965 T3681 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1290967 T3681 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1290978 T3681 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1290984 T3681 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1290995 T3681 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1291001 T3681 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1291001 T3681 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1291003 T3681 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1291006 T3681 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1291007 T3681 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1291008 T3681 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1291009 T3681 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty2/
[junit4] 2> 1291010 T3681 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2f99c27e
[junit4] 2> 1291015 T3681 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty2
[junit4] 2> 1291015 T3681 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty2/index/
[junit4] 2> 1291016 T3681 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 1291017 T3681 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty2/index
[junit4] 2> 1291021 T3681 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@a8d77bfa lockFactory=org.apache.lucene.store.NativeFSLockFactory@c4f0017),segFN=segments_1,generation=1}
[junit4] 2> 1291021 T3681 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1291033 T3681 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1291033 T3681 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1291035 T3681 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1291036 T3681 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1291037 T3681 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1291038 T3681 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1291039 T3681 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1291040 T3681 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1291040 T3681 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1291042 T3681 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1291046 T3681 oass.SolrIndexSearcher.<init> Opening Searcher@3afcff90 main
[junit4] 2> 1291049 T3681 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1291049 T3681 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1291055 T3682 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3afcff90 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1291057 T3681 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1291057 T3681 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38771/n_b/m collection:collection1 shard:shard2
[junit4] 2> 1291058 T3681 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 1291065 T3681 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1291073 T3681 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1291076 T3681 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1291076 T3681 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1291076 T3681 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38771/n_b/m/collection1/
[junit4] 2> 1291077 T3681 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1291077 T3681 oasc.SyncStrategy.syncToMe http://127.0.0.1:38771/n_b/m/collection1/ has no replicas
[junit4] 2> 1291078 T3681 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38771/n_b/m/collection1/ shard2
[junit4] 2> 1291078 T3681 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1292558 T3641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1292564 T3680 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1292564 T3660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1292564 T3640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1292564 T3647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1292607 T3681 oasc.ZkController.register We are http://127.0.0.1:38771/n_b/m/collection1/ and leader is http://127.0.0.1:38771/n_b/m/collection1/
[junit4] 2> 1292607 T3681 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38771/n_b/m
[junit4] 2> 1292607 T3681 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1292608 T3681 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1292608 T3681 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1292610 T3681 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1292611 T3618 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 1292612 T3618 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1292613 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1292723 T3618 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1292729 T3618 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57465
[junit4] 2> 1292731 T3618 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1292732 T3618 oasc.SolrResourceLoader.locateSolrHome JNDI
[...truncated too long message...]
un(QueuedThreadPool.java:543)
[junit4] > at java.lang.Thread.run(Thread.java:780)Throwable #15: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1024, name=HashSessionScavenger-17, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space
[junit4] > at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
[junit4] > at java.security.AccessController.toArrayOfProtectionDomains(AccessController.java:184)
[junit4] > at java.security.AccessController.checkPermission(AccessController.java:102)
[junit4] > at java.lang.SecurityManager.checkPermission(SecurityManager.java:560)
[junit4] > at java.lang.Thread.setContextClassLoader(Thread.java:797)
[junit4] > at org.eclipse.jetty.server.session.HashSessionManager.scavenge(HashSessionManager.java:344)
[junit4] > at org.eclipse.jetty.server.session.HashSessionManager$2.run(HashSessionManager.java:284)
[junit4] > at java.util.Timer$TimerImpl.run(Timer.java:296)Throwable #16: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=964, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space
[junit4] > at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
[junit4] > at java.util.ArrayList$SubList.listIterator(ArrayList.java:1024)
[junit4] > at java.util.AbstractList.listIterator(AbstractList.java:629)
[junit4] > at java.util.ArrayList$SubList.iterator(ArrayList.java:1016)
[junit4] > at java.util.AbstractCollection.toArray(AbstractCollection.java:363)
[junit4] > at java.util.regex.Pattern.split(Pattern.java:1238)
[junit4] > at java.util.regex.Pattern.split(Pattern.java:1270)
[junit4] > at java.lang.String.split(String.java:1749)
[junit4] > at org.apache.zookeeper.jmx.MBeanRegistry.tokenize(MBeanRegistry.java:182)
[junit4] > at org.apache.zookeeper.jmx.MBeanRegistry.makeObjectName(MBeanRegistry.java:204)
[junit4] > at org.apache.zookeeper.jmx.MBeanRegistry.unregister(MBeanRegistry.java:115)
[junit4] > at org.apache.zookeeper.jmx.MBeanRegistry.unregister(MBeanRegistry.java:132)
[junit4] > at org.apache.zookeeper.server.ServerCnxnFactory.unregisterConnection(ServerCnxnFactory.java:137)
[junit4] > at org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:972)
[junit4] > at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:352)
[junit4] > at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] > at java.lang.Thread.run(Thread.java:780)Throwable #17: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1000, name=qtp220461705-1000, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space
[junit4] > at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
[junit4] > at java.util.concurrent.ConcurrentHashMap$KeySet.iterator(ConcurrentHashMap.java:1273)
[junit4] > at org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1.run(SelectorManager.java:712)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] > at java.lang.Thread.run(Thread.java:780)Throwable #18: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1096, name=qtp1639013060-1096, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space
[junit4] > at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1868)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2084)
[junit4] > at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
[junit4] > at java.lang.Thread.run(Thread.java:780)Throwable #19: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1009, name=qtp1639013060-1009 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space
[junit4] > at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
[junit4] > at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:304)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] > at java.lang.Thread.run(Thread.java:780)Throwable #20: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1025, name=qtp-1726714643-1025, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space
[junit4] > at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
[junit4] > at java.util.HashMap$1.iterator(HashMap.java:672)
[junit4] > at java.util.HashSet.iterator(HashSet.java:173)
[junit4] > at sun.nio.ch.Util$2.iterator(Util.java:315)
[junit4] > at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
[junit4] > at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] > at java.lang.Thread.run(Thread.java:780)Throwable #21: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1209, name=qtp1556593299-1209, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1868)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2084)
[junit4] > at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
[junit4] > at java.lang.Thread.run(Thread.java:780)Throwable #22: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1041, name=HashSessionScavenger-18, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space
[junit4] > at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
[junit4] > at java.security.AccessController.getProtectionDomains(Native Method)
[junit4] > at java.security.AccessController.checkPermission(AccessController.java:88)
[junit4] > at java.lang.SecurityManager.checkPermission(SecurityManager.java:560)
[junit4] > at java.lang.Thread.setContextClassLoader(Thread.java:797)
[junit4] > at org.eclipse.jetty.server.session.HashSessionManager.scavenge(HashSessionManager.java:344)
[junit4] > at org.eclipse.jetty.server.session.HashSessionManager$2.run(HashSessionManager.java:284)
[junit4] > at java.util.Timer$TimerImpl.run(Timer.java:296)
[junit4] Completed on J1 in 7226.21s, 1 test, 23 errors <<< FAILURES!
[...truncated 229 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:389: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:382: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:450: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1230: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:873: There were test failures: 320 suites, 1360 tests, 22 suite-level errors, 2 errors, 215 ignored (5 assumptions)
Total time: 151 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b102) - Build
# 6998 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6998/
Java: 64bit/jdk1.8.0-ea-b102 -XX:+UseCompressedOops -XX:+UseG1GC
12 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([CA626772DFF8AF14]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1427, name=qtp2077582201-1427, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1427, name=qtp2077582201-1427, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1381, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1381, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1460, name=qtp2038018135-1460, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1460, name=qtp2038018135-1460, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1458, name=HashSessionScavenger-49, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1458, name=HashSessionScavenger-49, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1503, name=Thread-711, state=RUNNABLE, group=Overseer state updater.]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1503, name=Thread-711, state=RUNNABLE, group=Overseer state updater.]
Caused by: java.lang.OutOfMemoryError: Java heap space
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1426, name=qtp2077582201-1426, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1426, name=qtp2077582201-1426, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1382, name=SessionTracker, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1382, name=SessionTracker, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1411, name=qtp451236801-1411, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1411, name=qtp451236801-1411, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([CA626772DFF8AF14]:0)
at java.util.HashMap.newKeyIterator(HashMap.java:2472)
at java.util.HashMap$KeySet.iterator(HashMap.java:2506)
at java.util.HashSet.iterator(HashSet.java:170)
at sun.nio.ch.Util$2.iterator(Util.java:243)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1443, name=qtp1634913681-1443, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1443, name=qtp1634913681-1443, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([CA626772DFF8AF14]:0)
at java.util.HashMap.newKeyIterator(HashMap.java:2472)
at java.util.HashMap$KeySet.iterator(HashMap.java:2506)
at java.util.HashSet.iterator(HashSet.java:170)
at sun.nio.ch.Util$2.iterator(Util.java:243)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1442, name=qtp1634913681-1442 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1442, name=qtp1634913681-1442 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
FAILED: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([CA626772DFF8AF14]:0)
Build Log:
[...truncated 10368 lines...]
[junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
[junit4] 2> 322856 T1379 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 322859 T1379 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-FullSolrCloudDistribCmdsTest-1376409781697
[junit4] 2> 322860 T1379 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 322861 T1380 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 322961 T1379 oasc.ZkTestServer.run start zk server on port:40259
[junit4] 2> 322962 T1379 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 322996 T1386 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@714f8225 name:ZooKeeperConnection Watcher:127.0.0.1:40259 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 322996 T1379 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 322997 T1379 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 323016 T1379 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 323017 T1388 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f88727c name:ZooKeeperConnection Watcher:127.0.0.1:40259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 323018 T1379 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 323019 T1379 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 323021 T1379 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 323023 T1379 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 323024 T1379 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 323025 T1379 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 323026 T1379 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 323028 T1379 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 323028 T1379 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 323031 T1379 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 323031 T1379 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 323040 T1379 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 323041 T1379 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 323043 T1379 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 323044 T1379 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 323049 T1379 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 323049 T1379 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 323051 T1379 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 323051 T1379 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 323053 T1379 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 323053 T1379 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 323055 T1379 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 323055 T1379 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 323056 T1379 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 323057 T1379 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 323134 T1379 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 323137 T1379 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41202
[junit4] 2> 323138 T1379 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 323139 T1379 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 323139 T1379 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376409781897
[junit4] 2> 323139 T1379 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376409781897/'
[junit4] 2> 323150 T1379 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376409781897/solr.xml
[junit4] 2> 323159 T1379 oasc.CoreContainer.<init> New CoreContainer 90753564
[junit4] 2> 323159 T1379 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376409781897/]
[junit4] 2> 323160 T1379 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 323161 T1379 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 323161 T1379 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 323161 T1379 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 323162 T1379 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 323162 T1379 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 323162 T1379 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 323163 T1379 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 323163 T1379 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 323163 T1379 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 323168 T1379 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40259/solr
[junit4] 2> 323168 T1379 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 323169 T1379 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 323171 T1399 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7473a8d3 name:ZooKeeperConnection Watcher:127.0.0.1:40259 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 323171 T1379 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 323173 T1379 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 323177 T1379 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 323178 T1401 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48a5240a name:ZooKeeperConnection Watcher:127.0.0.1:40259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 323179 T1379 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 323180 T1379 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 323183 T1379 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 323185 T1379 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 323187 T1379 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41202_
[junit4] 2> 323188 T1379 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41202_
[junit4] 2> 323190 T1379 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 323194 T1379 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 323197 T1379 oasc.Overseer.start Overseer (id=90204391453491203-127.0.0.1:41202_-n_0000000000) starting
[junit4] 2> 323199 T1379 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 323201 T1379 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 323202 T1403 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 323203 T1379 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 323204 T1379 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 323206 T1402 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 323207 T1404 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 323207 T1404 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 323209 T1404 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 324709 T1402 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 324710 T1402 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:41202_",
[junit4] 2> "base_url":"http://127.0.0.1:41202"}
[junit4] 2> 324710 T1402 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 324710 T1402 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 324715 T1401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 325210 T1404 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 325210 T1404 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376409781897/collection1
[junit4] 2> 325210 T1404 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 325211 T1404 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 325211 T1404 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 325212 T1404 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376409781897/collection1/'
[junit4] 2> 325212 T1404 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376409781897/collection1/lib/README' to classloader
[junit4] 2> 325212 T1404 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376409781897/collection1/lib/classes/' to classloader
[junit4] 2> 325226 T1404 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 325233 T1404 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 325234 T1404 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 325239 T1404 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 325347 T1404 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 325352 T1404 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 325353 T1404 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 325362 T1404 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 325364 T1404 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 325366 T1404 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 325366 T1404 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 325367 T1404 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 325367 T1404 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 325367 T1404 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 325367 T1404 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 325368 T1404 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 325368 T1404 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376409781897/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/control/data/
[junit4] 2> 325368 T1404 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d346920
[junit4] 2> 325368 T1404 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/control/data
[junit4] 2> 325368 T1404 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/control/data/index/
[junit4] 2> 325369 T1404 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 325369 T1404 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/control/data/index
[junit4] 2> 325370 T1404 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6e7eeec3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4a8eb53),segFN=segments_1,generation=1}
[junit4] 2> 325370 T1404 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 325372 T1404 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 325372 T1404 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 325373 T1404 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 325373 T1404 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 325373 T1404 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 325373 T1404 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 325374 T1404 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 325374 T1404 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 325374 T1404 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 325375 T1404 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 325376 T1404 oass.SolrIndexSearcher.<init> Opening Searcher@16c9783c main
[junit4] 2> 325377 T1404 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 325377 T1404 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 325378 T1405 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@16c9783c main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 325378 T1404 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 325378 T1404 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41202 collection:control_collection shard:shard1
[junit4] 2> 325379 T1404 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 325381 T1404 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 325385 T1404 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 325387 T1404 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 325387 T1404 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 325387 T1404 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:41202/collection1/
[junit4] 2> 325387 T1404 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 325387 T1404 oasc.SyncStrategy.syncToMe http://127.0.0.1:41202/collection1/ has no replicas
[junit4] 2> 325387 T1404 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:41202/collection1/ shard1
[junit4] 2> 325388 T1404 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 326218 T1402 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 326223 T1401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 326248 T1404 oasc.ZkController.register We are http://127.0.0.1:41202/collection1/ and leader is http://127.0.0.1:41202/collection1/
[junit4] 2> 326248 T1404 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41202
[junit4] 2> 326249 T1404 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 326249 T1404 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 326249 T1404 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 326250 T1404 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 326251 T1379 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 326251 T1379 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 326251 T1379 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 326253 T1379 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4] 2> 326254 T1379 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 326258 T1408 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5381376e name:ZooKeeperConnection Watcher:127.0.0.1:40259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 326258 T1379 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 326259 T1379 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 326260 T1379 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4] 2> 326325 T1379 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 326327 T1379 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45314
[junit4] 2> 326328 T1379 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 326328 T1379 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 326328 T1379 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376409785097
[junit4] 2> 326328 T1379 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376409785097/'
[junit4] 2> 326338 T1379 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376409785097/solr.xml
[junit4] 2> 326345 T1379 oasc.CoreContainer.<init> New CoreContainer 99983877
[junit4] 2> 326345 T1379 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376409785097/]
[junit4] 2> 326346 T1379 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 326346 T1379 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 326346 T1379 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 326346 T1379 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 326346 T1379 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 326347 T1379 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 326347 T1379 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 326347 T1379 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 326347 T1379 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 326347 T1379 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 326350 T1379 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40259/solr
[junit4] 2> 326350 T1379 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 326351 T1379 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 326353 T1419 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@521ef3c1 name:ZooKeeperConnection Watcher:127.0.0.1:40259 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 326353 T1379 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 326354 T1379 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 326357 T1379 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 326358 T1421 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54893882 name:ZooKeeperConnection Watcher:127.0.0.1:40259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 326358 T1379 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 326360 T1379 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 327362 T1379 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45314_
[junit4] 2> 327363 T1379 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45314_
[junit4] 2> 327366 T1401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 327367 T1408 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 327367 T1421 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 327367 T1401 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 327370 T1422 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 327370 T1422 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 327371 T1422 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 327726 T1402 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 327727 T1402 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:41202_",
[junit4] 2> "base_url":"http://127.0.0.1:41202"}
[junit4] 2> 327729 T1402 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:45314_",
[junit4] 2> "base_url":"http://127.0.0.1:45314"}
[junit4] 2> 327730 T1402 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 327730 T1402 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 327732 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 327732 T1408 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 327732 T1401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 328371 T1422 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 328371 T1422 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376409785097/collection1
[junit4] 2> 328372 T1422 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 328372 T1422 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 328372 T1422 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 328373 T1422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376409785097/collection1/'
[junit4] 2> 328373 T1422 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376409785097/collection1/lib/README' to classloader
[junit4] 2> 328374 T1422 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376409785097/collection1/lib/classes/' to classloader
[junit4] 2> 328387 T1422 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 328394 T1422 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 328395 T1422 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 328401 T1422 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 328504 T1422 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 328507 T1422 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 328508 T1422 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 328515 T1422 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 328517 T1422 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 328519 T1422 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 328519 T1422 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 328519 T1422 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 328520 T1422 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 328520 T1422 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 328520 T1422 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 328520 T1422 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 328521 T1422 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376409785097/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/jetty1/
[junit4] 2> 328521 T1422 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d346920
[junit4] 2> 328521 T1422 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/jetty1
[junit4] 2> 328522 T1422 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/jetty1/index/
[junit4] 2> 328522 T1422 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 328522 T1422 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/jetty1/index
[junit4] 2> 328523 T1422 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5e45441 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5b9d2702),segFN=segments_1,generation=1}
[junit4] 2> 328523 T1422 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 328525 T1422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 328525 T1422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 328526 T1422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 328526 T1422 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 328526 T1422 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 328527 T1422 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 328527 T1422 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 328527 T1422 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 328527 T1422 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 328529 T1422 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 328529 T1422 oass.SolrIndexSearcher.<init> Opening Searcher@443e3e24 main
[junit4] 2> 328530 T1422 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 328530 T1422 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 328532 T1423 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@443e3e24 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 328533 T1422 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 328533 T1422 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45314 collection:collection1 shard:shard1
[junit4] 2> 328534 T1422 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 328537 T1422 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 328540 T1422 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 328542 T1422 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 328542 T1422 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 328542 T1422 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45314/collection1/
[junit4] 2> 328542 T1422 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 328542 T1422 oasc.SyncStrategy.syncToMe http://127.0.0.1:45314/collection1/ has no replicas
[junit4] 2> 328543 T1422 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45314/collection1/ shard1
[junit4] 2> 328543 T1422 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 329236 T1402 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 329242 T1408 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 329242 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 329242 T1401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 329248 T1422 oasc.ZkController.register We are http://127.0.0.1:45314/collection1/ and leader is http://127.0.0.1:45314/collection1/
[junit4] 2> 329249 T1422 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45314
[junit4] 2> 329249 T1422 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 329249 T1422 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 329249 T1422 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 329251 T1422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 329252 T1379 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 329252 T1379 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 329252 T1379 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 329313 T1379 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 329314 T1379 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49936
[junit4] 2> 329315 T1379 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 329315 T1379 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 329316 T1379 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376409788091
[junit4] 2> 329316 T1379 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376409788091/'
[junit4] 2> 329326 T1379 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376409788091/solr.xml
[junit4] 2> 329334 T1379 oasc.CoreContainer.<init> New CoreContainer 1680629742
[junit4] 2> 329334 T1379 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376409788091/]
[junit4] 2> 329335 T1379 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 329335 T1379 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 329335 T1379 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 329335 T1379 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 329336 T1379 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 329336 T1379 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 329336 T1379 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 329336 T1379 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 329337 T1379 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 329337 T1379 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 329340 T1379 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40259/solr
[junit4] 2> 329340 T1379 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 329341 T1379 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 329344 T1435 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72697b8 name:ZooKeeperConnection Watcher:127.0.0.1:40259 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 329344 T1379 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 329348 T1379 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 329351 T1379 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 329353 T1437 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6059bc00 name:ZooKeeperConnection Watcher:127.0.0.1:40259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 329353 T1379 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 329365 T1379 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 330368 T1379 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49936_
[junit4] 2> 330369 T1379 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49936_
[junit4] 2> 330371 T1401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 330372 T1437 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 330372 T1421 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 330372 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 330372 T1408 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 330372 T1408 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 330373 T1401 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 330374 T1438 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 330374 T1438 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 330375 T1438 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 330747 T1402 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 330748 T1402 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:45314_",
[junit4] 2> "base_url":"http://127.0.0.1:45314"}
[junit4] 2> 330750 T1402 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:49936_",
[junit4] 2> "base_url":"http://127.0.0.1:49936"}
[junit4] 2> 330750 T1402 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 330750 T1402 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 330756 T1408 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 330756 T1401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 330756 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 330756 T1437 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 331375 T1438 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 331376 T1438 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376409788091/collection1
[junit4] 2> 331376 T1438 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 331377 T1438 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 331377 T1438 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 331377 T1438 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376409788091/collection1/'
[junit4] 2> 331378 T1438 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376409788091/collection1/lib/README' to classloader
[junit4] 2> 331378 T1438 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376409788091/collection1/lib/classes/' to classloader
[junit4] 2> 331391 T1438 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 331398 T1438 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 331399 T1438 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 331404 T1438 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 331507 T1438 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 331510 T1438 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 331512 T1438 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 331519 T1438 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 331520 T1438 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 331522 T1438 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 331523 T1438 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 331523 T1438 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 331523 T1438 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 331523 T1438 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 331524 T1438 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 331524 T1438 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 331524 T1438 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376409788091/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/jetty2/
[junit4] 2> 331524 T1438 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d346920
[junit4] 2> 331525 T1438 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/jetty2
[junit4] 2> 331525 T1438 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/jetty2/index/
[junit4] 2> 331525 T1438 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 331525 T1438 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/jetty2/index
[junit4] 2> 331526 T1438 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6948c532 lockFactory=org.apache.lucene.store.NativeFSLockFactory@26456fa3),segFN=segments_1,generation=1}
[junit4] 2> 331526 T1438 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 331528 T1438 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 331529 T1438 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 331529 T1438 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 331529 T1438 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 331530 T1438 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 331530 T1438 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 331530 T1438 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 331530 T1438 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 331530 T1438 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 331532 T1438 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 331533 T1438 oass.SolrIndexSearcher.<init> Opening Searcher@1600c78d main
[junit4] 2> 331533 T1438 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 331533 T1438 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 331535 T1439 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1600c78d main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 331535 T1438 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 331535 T1438 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49936 collection:collection1 shard:shard2
[junit4] 2> 331536 T1438 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 331538 T1438 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 331542 T1438 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 331543 T1438 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 331543 T1438 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 331543 T1438 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49936/collection1/
[junit4] 2> 331544 T1438 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 331544 T1438 oasc.SyncStrategy.syncToMe http://127.0.0.1:49936/collection1/ has no replicas
[junit4] 2> 331544 T1438 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49936/collection1/ shard2
[junit4] 2> 331544 T1438 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 332259 T1402 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 332264 T1408 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 332264 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 332265 T1401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 332265 T1437 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 332298 T1438 oasc.ZkController.register We are http://127.0.0.1:49936/collection1/ and leader is http://127.0.0.1:49936/collection1/
[junit4] 2> 332299 T1438 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49936
[junit4] 2> 332299 T1438 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 332299 T1438 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 332299 T1438 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 332300 T1438 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 332301 T1379 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 332301 T1379 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 332302 T1379 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 332374 T1379 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 332376 T1379 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57730
[junit4] 2> 332376 T1379 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 332377 T1379 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 332377 T1379 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376409791141
[junit4] 2> 332377 T1379 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376409791141/'
[junit4] 2> 332387 T1379 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376409791141/solr.xml
[junit4] 2> 332394 T1379 oasc.CoreContainer.<init> New CoreContainer 40941163
[junit4] 2> 332395 T1379 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376409791141/]
[junit4] 2> 332395 T1379 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 332395 T1379 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 332396 T1379 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 332396 T1379 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 332396 T1379 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 332396 T1379 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 332397 T1379 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 332397 T1379 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 332397 T1379 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 332397 T1379 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 332400 T1379 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40259/solr
[junit4] 2> 332400 T1379 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 332401 T1379 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 332403 T1451 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75b5f50e name:ZooKeeperConnection Watcher:127.0.0.1:40259 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 332403 T1379 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 332404 T1379 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 332407 T1379 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 332408 T1453 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8d3a725 name:ZooKeeperConnection Watcher:127.0.0.1:40259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 332409 T1379 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 332411 T1379 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 333414 T1379 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57730_
[junit4] 2> 333415 T1379 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57730_
[junit4] 2> 333417 T1401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 333417 T1408 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 333418 T1408 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 333418 T1421 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 333418 T1453 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 333418 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 333418 T1437 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 333419 T1437 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 333419 T1401 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 333421 T1454 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 333421 T1454 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 333422 T1454 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 333769 T1402 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 333770 T1402 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard2",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:49936_",
[junit4] 2> "base_url":"http://127.0.0.1:49936"}
[junit4] 2> 333772 T1402 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:57730_",
[junit4] 2> "base_url":"http://127.0.0.1:57730"}
[junit4] 2> 333773 T1402 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 333773 T1402 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 333775 T1408 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 333775 T1437 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 333776 T1401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 333775 T1453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 333775 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 334422 T1454 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 334422 T1454 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376409791141/collection1
[junit4] 2> 334423 T1454 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 334424 T1454 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 334424 T1454 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 334425 T1454 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376409791141/collection1/'
[junit4] 2> 334426 T1454 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376409791141/collection1/lib/README' to classloader
[junit4] 2> 334427 T1454 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376409791141/collection1/lib/classes/' to classloader
[junit4] 2> 334444 T1454 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 334451 T1454 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 334453 T1454 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 334457 T1454 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 334545 T1454 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 334549 T1454 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 334550 T1454 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 334557 T1454 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 334559 T1454 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 334585 T1454 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 334586 T1454 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 334586 T1454 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 334587 T1454 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 334587 T1454 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 334587 T1454 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 334587 T1454 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 334588 T1454 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376409791141/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/jetty3/
[junit4] 2> 334588 T1454 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d346920
[junit4] 2> 334588 T1454 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/jetty3
[junit4] 2> 334588 T1454 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/jetty3/index/
[junit4] 2> 334589 T1454 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 334589 T1454 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376409781697/jetty3/index
[junit4] 2> 334590 T1454 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@43409f14 lockFactory=org.apache.lucene.store.NativeFSLockFactory@57f90120),segFN=segments_1,generation=1}
[junit4] 2> 334590 T1454 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 334592 T1454 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 334592 T1454 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 334593 T1454 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 334593 T1454 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 334593 T1454 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 334594 T1454 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 334594 T1454 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 334594 T1454 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 334594 T1454 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 334596 T1454 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 334596 T1454 oass.SolrIndexSearcher.<init> Opening Searcher@5504ff1c main
[junit4] 2> 334597 T1454 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 334597 T1454 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 334599 T1455 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5504ff1c main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 334600 T1454 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 334600 T1454 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57730 collection:collection1 shard:shard1
[junit4] 2> 334600 T1454 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 334605 T1454 oasc.ZkController.register We are http://127.0.0.1:57730/collection1/ and leader is http://127.0.0.1:45314/collection1/
[junit4] 2> 334605 T1454 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57730
[junit4] 2> 334605 T1454 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 334606 T1454 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C387 name=collection1 org.apache.solr.core.SolrCore@455d491c url=http://127.0.0.1:57730/collection1 node=127.0.0.1:57730_ C387_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:57730_, base_url=http://127.0.0.1:57730}
[junit4] 2> 334606 T1456 C387 P57730 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 334606 T1454 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 334607 T1456 C387 P57730 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 334607 T1456 C387 P57730 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4] 2> 334607 T1456 C387 P57730 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 334607 T1379 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 334608 T1379 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 334608 T1456 C387 P57730 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 334608 T1379 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 334612 T1416 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 334681 T1379 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 334684 T1379 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39994
[junit4] 2> 334684 T1379 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 334685 T1379 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 334685 T1379 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376409793447
[junit4] 2> 334685 T1379 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376409793447/'
[junit4] 2> 334699 T1379 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376409793447/solr.xml
[junit4] 2> 334706 T1379 oasc.CoreContainer.<init> New CoreContainer 1997155242
[junit4] 2> 334707 T1379 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376409793447/]
[junit4] 2> 334707 T1379 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 334707 T1379 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 334708 T1379 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 334708 T1379 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 334708 T1379 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 334708 T1379 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 334708 T1379 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 334709 T1379 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 334709 T1379 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 334709 T1379 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 334712 T1379 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40259/solr
[junit4] 2> 334712 T1379 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 334713 T1379 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 334714 T1468 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d4d6e2d name:ZooKeeperConnection Watcher:127.0.0.1:40259 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 334715 T1379 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 334716 T1379 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 334718 T1379 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 334720 T1470 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9bbc4e7 name:ZooKeeperConnection Watcher:127.0.0.1:40259/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 334720 T1379 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 334722 T1379 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 335279 T1402 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 335280 T1402 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"recovering",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:57730_",
[junit4] 2> "base_url":"http://127.0.0.1:57730"}
[junit4] 2> 335283 T1408 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 335283 T1470 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 335284 T1401 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 335283 T1421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 335283 T1437 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 335284 T1453 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 335612 T1416 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 335613 T1416 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node3&state=recovering&nodeName=127.0.0.1:57730_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTi
[...truncated too long message...]
ipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4] 2> 8) Thread[id=1441, name=HashSessionScavenger-48, state=TIMED_WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.util.TimerThread.mainLoop(Timer.java:552)
[junit4] 2> at java.util.TimerThread.run(Timer.java:505)
[junit4] 2> 9) Thread[id=1473, name=searcherExecutor-1030-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2037)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4] 2> 10) Thread[id=1462, name=qtp2038018135-1462 Acceptor1 SelectChannelConnector@127.0.0.1:39994, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] 2> at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)
[junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4] 2> NOTE: test params are: codec=Asserting, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=es_BO, timezone=Atlantic/Cape_Verde
[junit4] 2> NOTE: Linux 3.2.0-51-generic amd64/Oracle Corporation 1.8.0-ea (64-bit)/cpus=8,threads=11,free=207176640,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [XmlUpdateRequestHandlerTest, BadCopyFieldTest, SolrPluginUtilsTest, TestLazyCores, TestBinaryResponseWriter, BasicZkTest, RequestHandlersTest, TestDynamicFieldResource, ConvertedLegacyTest, TestMultiCoreConfBootstrap, ChaosMonkeySafeLeaderTest, OutputWriterTest, PrimUtilsTest, StatsComponentTest, UniqFieldsUpdateProcessorFactoryTest, UUIDFieldTest, TestFastOutputStream, TestCloudManagedSchema, TestSolr4Spatial, DisMaxRequestHandlerTest, PingRequestHandlerTest, NoCacheHeaderTest, TestSchemaSimilarityResource, HdfsCollectionsAPIDistributedZkTest, TestManagedSchema, TestBinaryField, DeleteShardTest, TestDistributedSearch, DateMathParserTest, TestFastWriter, DirectSolrSpellCheckerTest, TestUpdate, TestDistributedGrouping, RAMDirectoryFactoryTest, TestQueryTypes, MinimalSchemaTest, TestDFRSimilarityFactory, TestAddFieldRealTimeGet, TestRandomFaceting, PreAnalyzedFieldTest, TestSolrCoreProperties, TestInfoStreamLogging, TestWordDelimiterFilterFactory, TestRandomMergePolicy, TestSuggestSpellingConverter, TestRecovery, HdfsBasicDistributedZk2Test, HighlighterConfigTest, FullSolrCloudDistribCmdsTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=FullSolrCloudDistribCmdsTest -Dtests.seed=CA626772DFF8AF14 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_BO -Dtests.timezone=Atlantic/Cape_Verde -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J1 | FullSolrCloudDistribCmdsTest (suite) <<<
[junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4] > at __randomizedtesting.SeedInfo.seed([CA626772DFF8AF14]:0)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1427, name=qtp2077582201-1427, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1381, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1460, name=qtp2038018135-1460, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1458, name=HashSessionScavenger-49, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1503, name=Thread-711, state=RUNNABLE, group=Overseer state updater.]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1426, name=qtp2077582201-1426, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #8: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1382, name=SessionTracker, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #9: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1411, name=qtp451236801-1411, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space
[junit4] > at __randomizedtesting.SeedInfo.seed([CA626772DFF8AF14]:0)
[junit4] > at java.util.HashMap.newKeyIterator(HashMap.java:2472)
[junit4] > at java.util.HashMap$KeySet.iterator(HashMap.java:2506)
[junit4] > at java.util.HashSet.iterator(HashSet.java:170)
[junit4] > at sun.nio.ch.Util$2.iterator(Util.java:243)
[junit4] > at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
[junit4] > at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] > at java.lang.Thread.run(Thread.java:724)Throwable #10: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1443, name=qtp1634913681-1443, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space
[junit4] > at __randomizedtesting.SeedInfo.seed([CA626772DFF8AF14]:0)
[junit4] > at java.util.HashMap.newKeyIterator(HashMap.java:2472)
[junit4] > at java.util.HashMap$KeySet.iterator(HashMap.java:2506)
[junit4] > at java.util.HashSet.iterator(HashSet.java:170)
[junit4] > at sun.nio.ch.Util$2.iterator(Util.java:243)
[junit4] > at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
[junit4] > at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] > at java.lang.Thread.run(Thread.java:724)Throwable #11: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1442, name=qtp1634913681-1442 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space
[junit4] Completed on J1 in 7223.55s, 1 test, 12 errors <<< FAILURES!
[...truncated 111 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:389: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:369: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:450: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1230: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:873: There were test failures: 320 suites, 1360 tests, 11 suite-level errors, 1 error, 111 ignored (6 assumptions)
Total time: 147 minutes 29 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-ea-b102 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_25) - Build #
6997 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6997/
Java: 64bit/jdk1.7.0_25 -XX:-UseCompressedOops -XX:+UseParallelGC
4 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([83718ADB2B906D29]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1640, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1640, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at __randomizedtesting.SeedInfo.seed([83718ADB2B906D29]:0)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:183)
at java.lang.Thread.run(Thread.java:724)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1649, name=qtp1079735746-1649, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1649, name=qtp1079735746-1649, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
FAILED: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([83718ADB2B906D29]:0)
Build Log:
[...truncated 10464 lines...]
[junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
[junit4] 2> 381549 T1638 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 381552 T1638 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-FullSolrCloudDistribCmdsTest-1376397621299
[junit4] 2> 381553 T1638 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 381553 T1639 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 381653 T1638 oasc.ZkTestServer.run start zk server on port:60829
[junit4] 2> 381654 T1638 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 381702 T1645 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2cf034ac name:ZooKeeperConnection Watcher:127.0.0.1:60829 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 381702 T1638 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 381702 T1638 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 381707 T1638 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 381708 T1647 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5b795762 name:ZooKeeperConnection Watcher:127.0.0.1:60829/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 381708 T1638 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 381708 T1638 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 381711 T1638 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 381713 T1638 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 381714 T1638 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 381716 T1638 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 381716 T1638 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 381719 T1638 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 381720 T1638 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 381722 T1638 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 381723 T1638 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 381724 T1638 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 381725 T1638 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 381727 T1638 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 381727 T1638 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 381729 T1638 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 381729 T1638 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 381731 T1638 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 381731 T1638 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 381733 T1638 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 381733 T1638 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 381735 T1638 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 381735 T1638 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 381737 T1638 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 381737 T1638 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 381817 T1638 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 381820 T1638 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37631
[junit4] 2> 381820 T1638 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 381820 T1638 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 381821 T1638 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376397621486
[junit4] 2> 381821 T1638 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376397621486/'
[junit4] 2> 381830 T1638 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376397621486/solr.xml
[junit4] 2> 381872 T1638 oasc.CoreContainer.<init> New CoreContainer 1127103805
[junit4] 2> 381872 T1638 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376397621486/]
[junit4] 2> 381873 T1638 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 381873 T1638 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 381874 T1638 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 381874 T1638 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 381874 T1638 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 381874 T1638 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 381874 T1638 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 381875 T1638 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 381875 T1638 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 381875 T1638 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 381878 T1638 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60829/solr
[junit4] 2> 381879 T1638 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 381879 T1638 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 381881 T1658 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5cb4f976 name:ZooKeeperConnection Watcher:127.0.0.1:60829 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 381882 T1638 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 381883 T1638 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 381886 T1638 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 381887 T1660 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4228abeb name:ZooKeeperConnection Watcher:127.0.0.1:60829/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 381887 T1638 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 381888 T1638 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 381893 T1638 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 381895 T1638 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 381896 T1638 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37631_
[junit4] 2> 381897 T1638 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37631_
[junit4] 2> 381899 T1638 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 381902 T1638 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 381903 T1638 oasc.Overseer.start Overseer (id=90203594509582339-127.0.0.1:37631_-n_0000000000) starting
[junit4] 2> 381905 T1638 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 381908 T1662 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 381911 T1638 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 381916 T1638 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 381918 T1638 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 381920 T1661 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 381924 T1663 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 381924 T1663 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 381925 T1663 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 383423 T1661 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 383424 T1661 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:37631_",
[junit4] 2> "base_url":"http://127.0.0.1:37631"}
[junit4] 2> 383424 T1661 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 383424 T1661 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 383429 T1660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 383925 T1663 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 383925 T1663 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376397621486/collection1
[junit4] 2> 383925 T1663 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 383926 T1663 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 383926 T1663 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 383927 T1663 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376397621486/collection1/'
[junit4] 2> 383928 T1663 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376397621486/collection1/lib/README' to classloader
[junit4] 2> 383928 T1663 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376397621486/collection1/lib/classes/' to classloader
[junit4] 2> 383964 T1663 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 384001 T1663 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 384003 T1663 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 384009 T1663 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 384389 T1663 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 384394 T1663 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 384397 T1663 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 384407 T1663 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 384410 T1663 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 384413 T1663 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 384414 T1663 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 384414 T1663 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 384414 T1663 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 384415 T1663 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 384415 T1663 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 384415 T1663 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 384416 T1663 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376397621486/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/control/data/
[junit4] 2> 384416 T1663 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@58ef38f0
[junit4] 2> 384416 T1663 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/control/data
[junit4] 2> 384417 T1663 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/control/data/index/
[junit4] 2> 384417 T1663 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 384417 T1663 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/control/data/index
[junit4] 2> 384418 T1663 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5d5489a9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@777a9064),segFN=segments_1,generation=1}
[junit4] 2> 384419 T1663 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 384421 T1663 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 384421 T1663 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 384422 T1663 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 384422 T1663 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 384422 T1663 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 384422 T1663 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 384423 T1663 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 384423 T1663 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 384423 T1663 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 384425 T1663 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 384428 T1663 oass.SolrIndexSearcher.<init> Opening Searcher@64f9644e main
[junit4] 2> 384428 T1663 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 384429 T1663 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 384431 T1664 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@64f9644e main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 384431 T1663 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 384432 T1663 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37631 collection:control_collection shard:shard1
[junit4] 2> 384432 T1663 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 384435 T1663 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 384439 T1663 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 384441 T1663 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 384442 T1663 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 384442 T1663 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:37631/collection1/
[junit4] 2> 384442 T1663 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 384442 T1663 oasc.SyncStrategy.syncToMe http://127.0.0.1:37631/collection1/ has no replicas
[junit4] 2> 384442 T1663 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:37631/collection1/ shard1
[junit4] 2> 384442 T1663 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 384932 T1661 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 384937 T1660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 384948 T1663 oasc.ZkController.register We are http://127.0.0.1:37631/collection1/ and leader is http://127.0.0.1:37631/collection1/
[junit4] 2> 384949 T1663 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37631
[junit4] 2> 384949 T1663 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 384949 T1663 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 384949 T1663 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 384951 T1663 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 384952 T1638 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 384952 T1638 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 384953 T1638 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 384956 T1638 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4] 2> 384957 T1638 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 384959 T1667 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3859adee name:ZooKeeperConnection Watcher:127.0.0.1:60829/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 384959 T1638 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 384960 T1638 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 384962 T1638 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4] 2> 385057 T1638 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 385059 T1638 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39593
[junit4] 2> 385060 T1638 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 385060 T1638 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 385060 T1638 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376397624709
[junit4] 2> 385061 T1638 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376397624709/'
[junit4] 2> 385075 T1638 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376397624709/solr.xml
[junit4] 2> 385118 T1638 oasc.CoreContainer.<init> New CoreContainer 1023998742
[junit4] 2> 385118 T1638 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376397624709/]
[junit4] 2> 385120 T1638 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 385120 T1638 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 385120 T1638 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 385120 T1638 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 385121 T1638 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 385121 T1638 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 385121 T1638 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 385121 T1638 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 385122 T1638 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 385122 T1638 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 385125 T1638 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60829/solr
[junit4] 2> 385125 T1638 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 385126 T1638 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 385127 T1678 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3027ef39 name:ZooKeeperConnection Watcher:127.0.0.1:60829 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 385127 T1638 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 385129 T1638 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 385132 T1638 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 385133 T1680 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e9d2736 name:ZooKeeperConnection Watcher:127.0.0.1:60829/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 385133 T1638 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 385137 T1638 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 386139 T1638 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39593_
[junit4] 2> 386140 T1638 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39593_
[junit4] 2> 386143 T1667 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 386144 T1660 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 386144 T1660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 386144 T1680 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 386150 T1681 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 386150 T1681 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 386151 T1681 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 386442 T1661 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 386443 T1661 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:37631_",
[junit4] 2> "base_url":"http://127.0.0.1:37631"}
[junit4] 2> 386446 T1661 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:39593_",
[junit4] 2> "base_url":"http://127.0.0.1:39593"}
[junit4] 2> 386446 T1661 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 386446 T1661 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 386449 T1667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 386449 T1660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 386449 T1680 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 387151 T1681 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 387152 T1681 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376397624709/collection1
[junit4] 2> 387152 T1681 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 387153 T1681 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 387153 T1681 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 387154 T1681 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376397624709/collection1/'
[junit4] 2> 387154 T1681 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376397624709/collection1/lib/README' to classloader
[junit4] 2> 387155 T1681 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376397624709/collection1/lib/classes/' to classloader
[junit4] 2> 387179 T1681 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 387209 T1681 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 387210 T1681 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 387216 T1681 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 387534 T1681 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 387543 T1681 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 387546 T1681 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 387558 T1681 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 387562 T1681 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 387565 T1681 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 387567 T1681 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 387567 T1681 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 387567 T1681 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 387568 T1681 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 387568 T1681 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 387569 T1681 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 387569 T1681 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376397624709/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/jetty1/
[junit4] 2> 387569 T1681 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@58ef38f0
[junit4] 2> 387570 T1681 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/jetty1
[junit4] 2> 387570 T1681 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/jetty1/index/
[junit4] 2> 387571 T1681 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 387571 T1681 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/jetty1/index
[junit4] 2> 387573 T1681 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7bc2a76a lockFactory=org.apache.lucene.store.NativeFSLockFactory@7ac4b5d4),segFN=segments_1,generation=1}
[junit4] 2> 387573 T1681 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 387578 T1681 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 387578 T1681 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 387579 T1681 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 387580 T1681 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 387580 T1681 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 387580 T1681 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 387581 T1681 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 387581 T1681 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 387582 T1681 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 387586 T1681 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 387589 T1681 oass.SolrIndexSearcher.<init> Opening Searcher@52ade616 main
[junit4] 2> 387590 T1681 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 387590 T1681 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 387594 T1682 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@52ade616 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 387595 T1681 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 387595 T1681 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39593 collection:collection1 shard:shard1
[junit4] 2> 387596 T1681 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 387599 T1681 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 387604 T1681 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 387606 T1681 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 387606 T1681 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 387606 T1681 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39593/collection1/
[junit4] 2> 387606 T1681 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 387606 T1681 oasc.SyncStrategy.syncToMe http://127.0.0.1:39593/collection1/ has no replicas
[junit4] 2> 387607 T1681 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39593/collection1/ shard1
[junit4] 2> 387607 T1681 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 387954 T1661 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 387963 T1667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 387963 T1680 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 387963 T1660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 387981 T1681 oasc.ZkController.register We are http://127.0.0.1:39593/collection1/ and leader is http://127.0.0.1:39593/collection1/
[junit4] 2> 387982 T1681 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39593
[junit4] 2> 387982 T1681 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 387982 T1681 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 387982 T1681 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 387984 T1681 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 387985 T1638 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 387986 T1638 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 387986 T1638 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 388085 T1638 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 388088 T1638 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37559
[junit4] 2> 388088 T1638 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 388089 T1638 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 388089 T1638 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376397627734
[junit4] 2> 388090 T1638 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376397627734/'
[junit4] 2> 388104 T1638 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376397627734/solr.xml
[junit4] 2> 388144 T1638 oasc.CoreContainer.<init> New CoreContainer 1182590131
[junit4] 2> 388145 T1638 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376397627734/]
[junit4] 2> 388146 T1638 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 388146 T1638 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 388146 T1638 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 388147 T1638 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 388147 T1638 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 388147 T1638 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 388147 T1638 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 388148 T1638 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 388148 T1638 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 388148 T1638 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 388151 T1638 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60829/solr
[junit4] 2> 388152 T1638 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 388152 T1638 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 388154 T1694 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7abdf0ff name:ZooKeeperConnection Watcher:127.0.0.1:60829 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 388154 T1638 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 388156 T1638 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 388159 T1638 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 388160 T1696 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ab38c3 name:ZooKeeperConnection Watcher:127.0.0.1:60829/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 388160 T1638 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 388164 T1638 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 389167 T1638 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37559_
[junit4] 2> 389168 T1638 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37559_
[junit4] 2> 389170 T1667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 389171 T1660 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 389171 T1660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 389171 T1696 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 389171 T1680 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 389172 T1680 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 389173 T1667 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 389177 T1697 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 389177 T1697 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 389178 T1697 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 389467 T1661 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 389468 T1661 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:39593_",
[junit4] 2> "base_url":"http://127.0.0.1:39593"}
[junit4] 2> 389471 T1661 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:37559_",
[junit4] 2> "base_url":"http://127.0.0.1:37559"}
[junit4] 2> 389471 T1661 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 389471 T1661 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 389534 T1667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 389534 T1680 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 389534 T1696 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 389534 T1660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 390178 T1697 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 390179 T1697 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376397627734/collection1
[junit4] 2> 390179 T1697 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 390179 T1697 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 390180 T1697 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 390180 T1697 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376397627734/collection1/'
[junit4] 2> 390181 T1697 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376397627734/collection1/lib/README' to classloader
[junit4] 2> 390181 T1697 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376397627734/collection1/lib/classes/' to classloader
[junit4] 2> 390207 T1697 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 390239 T1697 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 390240 T1697 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 390247 T1697 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 390563 T1697 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 390569 T1697 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 390571 T1697 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 390580 T1697 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 390583 T1697 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 390586 T1697 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 390587 T1697 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 390587 T1697 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 390587 T1697 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 390588 T1697 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 390588 T1697 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 390588 T1697 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 390588 T1697 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376397627734/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/jetty2/
[junit4] 2> 390588 T1697 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@58ef38f0
[junit4] 2> 390589 T1697 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/jetty2
[junit4] 2> 390589 T1697 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/jetty2/index/
[junit4] 2> 390589 T1697 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 390590 T1697 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/jetty2/index
[junit4] 2> 390591 T1697 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@400d7fbb lockFactory=org.apache.lucene.store.NativeFSLockFactory@7c55f31a),segFN=segments_1,generation=1}
[junit4] 2> 390591 T1697 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 390594 T1697 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 390594 T1697 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 390595 T1697 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 390595 T1697 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 390596 T1697 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 390596 T1697 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 390596 T1697 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 390597 T1697 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 390597 T1697 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 390599 T1697 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 390601 T1697 oass.SolrIndexSearcher.<init> Opening Searcher@13dc4803 main
[junit4] 2> 390602 T1697 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 390602 T1697 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 390605 T1698 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13dc4803 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 390607 T1697 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 390607 T1697 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37559 collection:collection1 shard:shard2
[junit4] 2> 390607 T1697 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 390610 T1697 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 390615 T1697 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 390617 T1697 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 390617 T1697 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 390617 T1697 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:37559/collection1/
[junit4] 2> 390617 T1697 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 390617 T1697 oasc.SyncStrategy.syncToMe http://127.0.0.1:37559/collection1/ has no replicas
[junit4] 2> 390617 T1697 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:37559/collection1/ shard2
[junit4] 2> 390617 T1697 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 391038 T1661 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 391045 T1667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 391045 T1660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 391045 T1680 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 391045 T1696 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 391072 T1697 oasc.ZkController.register We are http://127.0.0.1:37559/collection1/ and leader is http://127.0.0.1:37559/collection1/
[junit4] 2> 391072 T1697 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37559
[junit4] 2> 391072 T1697 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 391073 T1697 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 391073 T1697 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 391074 T1697 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 391075 T1638 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 391076 T1638 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 391076 T1638 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 391158 T1638 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 391160 T1638 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53531
[junit4] 2> 391160 T1638 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 391161 T1638 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 391161 T1638 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376397630825
[junit4] 2> 391161 T1638 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376397630825/'
[junit4] 2> 391171 T1638 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376397630825/solr.xml
[junit4] 2> 391200 T1638 oasc.CoreContainer.<init> New CoreContainer 1923809468
[junit4] 2> 391200 T1638 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376397630825/]
[junit4] 2> 391201 T1638 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 391201 T1638 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 391202 T1638 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 391202 T1638 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 391202 T1638 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 391202 T1638 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 391203 T1638 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 391203 T1638 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 391203 T1638 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 391203 T1638 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 391206 T1638 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60829/solr
[junit4] 2> 391206 T1638 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 391207 T1638 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 391208 T1710 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@747abb3c name:ZooKeeperConnection Watcher:127.0.0.1:60829 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 391209 T1638 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 391211 T1638 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 391214 T1638 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 391215 T1712 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d02ce83 name:ZooKeeperConnection Watcher:127.0.0.1:60829/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 391215 T1638 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 391218 T1638 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 392221 T1638 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53531_
[junit4] 2> 392222 T1638 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53531_
[junit4] 2> 392223 T1667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 392224 T1696 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 392224 T1660 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 392224 T1680 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 392225 T1712 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 392225 T1660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 392225 T1680 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 392226 T1696 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 392226 T1667 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 392230 T1713 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 392231 T1713 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 392232 T1713 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 392551 T1661 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 392553 T1661 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard2",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:37559_",
[junit4] 2> "base_url":"http://127.0.0.1:37559"}
[junit4] 2> 392556 T1661 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:53531_",
[junit4] 2> "base_url":"http://127.0.0.1:53531"}
[junit4] 2> 392556 T1661 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 392556 T1661 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 392559 T1667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 392559 T1696 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 392559 T1660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 392559 T1712 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 392559 T1680 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 393232 T1713 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 393232 T1713 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376397630825/collection1
[junit4] 2> 393232 T1713 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 393233 T1713 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 393233 T1713 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 393234 T1713 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376397630825/collection1/'
[junit4] 2> 393235 T1713 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376397630825/collection1/lib/README' to classloader
[junit4] 2> 393235 T1713 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376397630825/collection1/lib/classes/' to classloader
[junit4] 2> 393260 T1713 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 393288 T1713 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 393289 T1713 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 393295 T1713 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 393587 T1713 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 393592 T1713 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 393594 T1713 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 393604 T1713 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 393607 T1713 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 393610 T1713 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 393610 T1713 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 393611 T1713 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 393611 T1713 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 393611 T1713 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 393612 T1713 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 393612 T1713 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 393612 T1713 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376397630825/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/jetty3/
[junit4] 2> 393612 T1713 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@58ef38f0
[junit4] 2> 393613 T1713 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/jetty3
[junit4] 2> 393613 T1713 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/jetty3/index/
[junit4] 2> 393613 T1713 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 393613 T1713 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376397621299/jetty3/index
[junit4] 2> 393615 T1713 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@35006fa3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4df18717),segFN=segments_1,generation=1}
[junit4] 2> 393615 T1713 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 393618 T1713 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 393618 T1713 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 393618 T1713 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 393619 T1713 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 393619 T1713 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 393619 T1713 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 393619 T1713 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 393620 T1713 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 393620 T1713 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 393622 T1713 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 393624 T1713 oass.SolrIndexSearcher.<init> Opening Searcher@bef0e4f main
[junit4] 2> 393625 T1713 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 393625 T1713 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 393628 T1714 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@bef0e4f main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 393630 T1713 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 393630 T1713 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53531 collection:collection1 shard:shard1
[junit4] 2> 393630 T1713 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 393635 T1713 oasc.ZkController.register We are http://127.0.0.1:53531/collection1/ and leader is http://127.0.0.1:39593/collection1/
[junit4] 2> 393635 T1713 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53531
[junit4] 2> 393635 T1713 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 393635 T1713 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C523 name=collection1 org.apache.solr.core.SolrCore@132e86ed url=http://127.0.0.1:53531/collection1 node=127.0.0.1:53531_ C523_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:53531_, base_url=http://127.0.0.1:53531}
[junit4] 2> 393636 T1715 C523 P53531 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 393636 T1713 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 393636 T1715 C523 P53531 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 393637 T1715 C523 P53531 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4] 2> 393637 T1715 C523 P53531 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 393637 T1638 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 393637 T1638 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 393638 T1715 C523 P53531 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 393638 T1638 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 393642 T1674 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 393703 T1638 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 393705 T1638 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43935
[junit4] 2> 393705 T1638 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 393705 T1638 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 393706 T1638 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376397633387
[junit4] 2> 393706 T1638 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376397633387/'
[junit4] 2> 393715 T1638 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376397633387/solr.xml
[junit4] 2> 393745 T1638 oasc.CoreContainer.<init> New CoreContainer 1995361181
[junit4] 2> 393745 T1638 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376397633387/]
[junit4] 2> 393746 T1638 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 393746 T1638 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 393746 T1638 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 393747 T1638 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 393747 T1638 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 393747 T1638 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 393747 T1638 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 393748 T1638 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 393748 T1638 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 393748 T1638 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 393751 T1638 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:60829/solr
[junit4] 2> 393751 T1638 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 393752 T1638 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 393753 T1727 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@850d6cd name:ZooKeeperConnection Watcher:127.0.0.1:60829 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 393753 T1638 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 393755 T1638 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 393759 T1638 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 393760 T1729 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51e4992e name:ZooKeeperConnection Watcher:127.0.0.1:60829/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 393760 T1638 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 393763 T1638 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 394066 T1661 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 394067 T1661 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"recovering",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:53531_",
[junit4] 2> "base_url":"http://127.0.0.1:53531"}
[junit4] 2> 394070 T1712 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 394070 T1696 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 394070 T1667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 394070 T1660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 394070 T1729 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 394070 T1680 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 394642 T1674 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 394643 T1674 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node3&state=recovering&nodeName=127.0.0.1:53531_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4] 2> 394766 T1638 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43935_
[junit4] 2> 394767 T1638 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43935_
[junit4] 2> 394769 T1667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 394769 T1729 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 394769 T1696 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 394769 T1712 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 394769 T1660 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 394769 T1712 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4] 2> 394770 T1680 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 394770 T1680 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4] 2> 394770 T1660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4] 2> 394771 T1696 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 394771 T1667 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 394771 T1729 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 394776 T1730 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 394776 T1730 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 394777 T1730 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 395576 T1661 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 395578 T1661 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node4",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:43935_",
[junit4] 2> "base_url":"http://127.0.0.1:43935"}
[junit4] 2> 395578 T1661 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 395578 T1661 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 395582 T1712 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 395582 T1680 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 395582 T1660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 395582 T1729 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 395582 T1696 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 395582 T1667 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4] 2> 395777 T1730 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 395777 T1730 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376397633387/collection1
[junit4] 2> 395777 T1730 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 395778 T1730 oasc.ZkController.createCollectionZkNode Collection zk
[...truncated too long message...]
il.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4] 2> 36) Thread[id=1684, name=HashSessionScavenger-59, state=TIMED_WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.util.TimerThread.mainLoop(Timer.java:552)
[junit4] 2> at java.util.TimerThread.run(Timer.java:505)
[junit4] 2> 37) Thread[id=1700, name=HashSessionScavenger-60, state=TIMED_WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.util.TimerThread.mainLoop(Timer.java:552)
[junit4] 2> at java.util.TimerThread.run(Timer.java:505)
[junit4] 2> 38) Thread[id=1704, name=qtp1361015859-1704 Acceptor1 SelectChannelConnector@127.0.0.1:53531, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:956)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4] 2> 39) Thread[id=1659, name=TEST-FullSolrCloudDistribCmdsTest.testDistribSearch-seed#[83718ADB2B906D29]-SendThread(localhost.localdomain:60829), state=TIMED_WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:86)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:937)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:993)
[junit4] 2> 40) Thread[id=1672, name=qtp290512899-1672 Acceptor1 SelectChannelConnector@127.0.0.1:39593, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:956)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4] 2> 41) Thread[id=1670, name=qtp290512899-1670 Selector1, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4] 2> at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4] 2> 7606968 T1659 oaz.ClientCnxn$SendThread.run WARN Session 0x14077b0f8350003 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
[junit4] 2> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4] 2> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
[junit4] 2>
[junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=PostingsFormat(name=Memory doPackFST= true), n_ti1=Lucene41(blocksize=128), n_f1=PostingsFormat(name=SimpleText), text=Pulsing41(freqCutoff=11 minBlockSize=37 maxBlockSize=125), _version_=PostingsFormat(name=SimpleText), n_tl1=PostingsFormat(name=SimpleText), n_d1=PostingsFormat(name=Memory doPackFST= true), rnd_b=Lucene41(blocksize=128), intDefault=PostingsFormat(name=SimpleText), n_td1=PostingsFormat(name=SimpleText), timestamp=PostingsFormat(name=Memory doPackFST= true), id=Lucene41(blocksize=128), range_facet_sl=Lucene41(blocksize=128), a_t=PostingsFormat(name=Memory doPackFST= true), n_tdt1=PostingsFormat(name=SimpleText), range_facet_si=PostingsFormat(name=Memory doPackFST= true), n_l1=PostingsFormat(name=Memory doPackFST= true), multiDefault=PostingsFormat(name=Memory doPackFST= true), n_tf1=PostingsFormat(name=Memory doPackFST= true), n_dt1=PostingsFormat(name=SimpleText)}, docValues:{timestamp=DocValuesFormat(name=Lucene42)}, sim=DefaultSimilarity, locale=da, timezone=America/Montreal
[junit4] 2> NOTE: Linux 3.2.0-51-generic amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=8,threads=48,free=72737424,total=477233152
[junit4] 2> NOTE: All tests run in this JVM: [TestCSVResponseWriter, TestIndexingPerformance, StandardRequestHandlerTest, PluginInfoTest, SpellingQueryConverterTest, ClusterStateTest, TestDynamicFieldResource, TestSchemaVersionResource, UUIDFieldTest, TestAtomicUpdateErrorCases, TestUpdate, DirectSolrSpellCheckerTest, TestComponentsName, IndexBasedSpellCheckerTest, TestZkChroot, DisMaxRequestHandlerTest, TestSolrQueryParserResource, ResourceLoaderTest, TestPartialUpdateDeduplication, BasicZkTest, TestClassNameShortening, SuggesterTSTTest, CoreContainerCoreInitFailuresTest, TestCloudManagedSchema, SuggesterWFSTTest, OutputWriterTest, SignatureUpdateProcessorFactoryTest, SpellCheckCollatorTest, TestManagedSchema, CustomCollectionTest, TestIndexSearcher, HdfsBasicDistributedZk2Test, PreAnalyzedUpdateProcessorTest, IndexSchemaRuntimeFieldTest, DefaultValueUpdateProcessorTest, NotRequiredUniqueKeyTest, SolrPluginUtilsTest, TestCollationField, RequestHandlersTest, FieldAnalysisRequestHandlerTest, TestStressRecovery, TestBinaryResponseWriter, TestJmxIntegration, XmlUpdateRequestHandlerTest, XsltUpdateRequestHandlerTest, TestCSVLoader, TestSchemaSimilarityResource, TestCopyFieldCollectionResource, TestFiltering, TestFastWriter, TestDistributedSearch, SpellCheckComponentTest, TestIBSimilarityFactory, TestCodecSupport, TestRandomMergePolicy, TestInfoStreamLogging, SolrCmdDistributorTest, TestRangeQuery, TermVectorComponentDistributedTest, TestOmitPositions, HighlighterConfigTest, BadCopyFieldTest, TestNumberUtils, TestXIncludeConfig, TestFastOutputStream, TestPluginEnable, DirectSolrConnectionTest, DebugComponentTest, BadIndexSchemaTest, TestDistributedGrouping, TestManagedSchemaFieldResource, PreAnalyzedFieldTest, TestUtils, HighlighterTest, FullSolrCloudDistribCmdsTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=FullSolrCloudDistribCmdsTest -Dtests.seed=83718ADB2B906D29 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=da -Dtests.timezone=America/Montreal -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J1 | FullSolrCloudDistribCmdsTest (suite) <<<
[junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4] > at __randomizedtesting.SeedInfo.seed([83718ADB2B906D29]:0)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1640, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
[junit4] > at __randomizedtesting.SeedInfo.seed([83718ADB2B906D29]:0)
[junit4] > at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:183)
[junit4] > at java.lang.Thread.run(Thread.java:724)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1649, name=qtp1079735746-1649, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
[junit4] Completed on J1 in 7225.45s, 1 test, 4 errors <<< FAILURES!
[...truncated 29 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:389: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:369: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:450: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1230: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:873: There were test failures: 320 suites, 1360 tests, 3 suite-level errors, 1 error, 40 ignored (6 assumptions)
Total time: 144 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_25 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_25) - Build #
6996 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6996/
Java: 64bit/jdk1.7.0_25 -XX:-UseCompressedOops -XX:+UseParallelGC
1 tests failed.
FAILED: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch
Error Message:
Server at http://127.0.0.1:34548/ov_v/collection1 returned non ok status:500, message:Server Error
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:34548/ov_v/collection1 returned non ok status:500, message:Server Error
at __randomizedtesting.SeedInfo.seed([AC2C812CF5D193D:8B24460AB8027901]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexQueryDeleteHierarchical(FullSolrCloudDistribCmdsTest.java:259)
at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:124)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:724)
Build Log:
[...truncated 9861 lines...]
[junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
[junit4] 2> 924844 T3063 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ov_v/
[junit4] 2> 924848 T3063 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-FullSolrCloudDistribCmdsTest-1376392776073
[junit4] 2> 924848 T3063 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 924849 T3064 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 924949 T3063 oasc.ZkTestServer.run start zk server on port:36230
[junit4] 2> 924950 T3063 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 925149 T3070 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39ce02f7 name:ZooKeeperConnection Watcher:127.0.0.1:36230 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 925150 T3063 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 925150 T3063 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 925154 T3063 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 925155 T3072 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22a29f97 name:ZooKeeperConnection Watcher:127.0.0.1:36230/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 925155 T3063 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 925155 T3063 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 925157 T3063 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 925158 T3063 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 925159 T3063 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 925161 T3063 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 925161 T3063 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 925163 T3063 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 925164 T3063 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 925165 T3063 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 925166 T3063 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 925167 T3063 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 925167 T3063 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 925169 T3063 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 925169 T3063 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 925171 T3063 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 925171 T3063 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 925172 T3063 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 925173 T3063 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 925174 T3063 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 925174 T3063 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 925176 T3063 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 925176 T3063 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 925178 T3063 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 925178 T3063 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 925240 T3063 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 925254 T3063 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34548
[junit4] 2> 925255 T3063 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 925255 T3063 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 925255 T3063 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376392776405
[junit4] 2> 925255 T3063 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376392776405/'
[junit4] 2> 925273 T3063 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376392776405/solr.xml
[junit4] 2> 925303 T3063 oasc.CoreContainer.<init> New CoreContainer 146594956
[junit4] 2> 925304 T3063 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376392776405/]
[junit4] 2> 925305 T3063 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 925305 T3063 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 925305 T3063 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 925305 T3063 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 925306 T3063 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 925306 T3063 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 925306 T3063 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 925306 T3063 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 925306 T3063 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 925307 T3063 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 925310 T3063 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36230/solr
[junit4] 2> 925311 T3063 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 925311 T3063 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 925313 T3083 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7af331e4 name:ZooKeeperConnection Watcher:127.0.0.1:36230 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 925313 T3063 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 925314 T3063 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 925316 T3063 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 925317 T3085 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c3ccd9 name:ZooKeeperConnection Watcher:127.0.0.1:36230/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 925317 T3063 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 925318 T3063 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 925321 T3063 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 925322 T3063 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 925323 T3063 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34548_ov_v
[junit4] 2> 925324 T3063 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34548_ov_v
[junit4] 2> 925325 T3063 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 925328 T3063 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 925329 T3063 oasc.Overseer.start Overseer (id=90203276972916739-127.0.0.1:34548_ov_v-n_0000000000) starting
[junit4] 2> 925330 T3063 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 925332 T3087 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 925333 T3063 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 925334 T3063 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 925335 T3063 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 925336 T3086 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 925339 T3088 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 925339 T3088 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 925340 T3088 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 926838 T3086 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 926839 T3086 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:34548_ov_v",
[junit4] 2> "base_url":"http://127.0.0.1:34548/ov_v"}
[junit4] 2> 926839 T3086 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 926839 T3086 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 926843 T3085 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 927340 T3088 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 927341 T3088 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376392776405/collection1
[junit4] 2> 927341 T3088 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 927341 T3088 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 927341 T3088 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 927342 T3088 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376392776405/collection1/'
[junit4] 2> 927343 T3088 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376392776405/collection1/lib/README' to classloader
[junit4] 2> 927343 T3088 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376392776405/collection1/lib/classes/' to classloader
[junit4] 2> 927368 T3088 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 927398 T3088 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 927399 T3088 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 927404 T3088 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 927705 T3088 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 927709 T3088 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 927711 T3088 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 927719 T3088 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 927721 T3088 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 927724 T3088 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 927724 T3088 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 927724 T3088 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 927725 T3088 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 927725 T3088 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 927725 T3088 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 927725 T3088 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 927726 T3088 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376392776405/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/control/data/
[junit4] 2> 927726 T3088 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7d9acbbd
[junit4] 2> 927726 T3088 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/control/data
[junit4] 2> 927727 T3088 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/control/data/index/
[junit4] 2> 927727 T3088 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 927727 T3088 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/control/data/index
[junit4] 2> 927728 T3088 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@29b9afde lockFactory=org.apache.lucene.store.NativeFSLockFactory@37c85ee6),segFN=segments_1,generation=1}
[junit4] 2> 927729 T3088 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 927731 T3088 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 927731 T3088 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 927732 T3088 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 927732 T3088 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 927732 T3088 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 927733 T3088 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 927733 T3088 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 927733 T3088 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 927733 T3088 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 927735 T3088 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 927737 T3088 oass.SolrIndexSearcher.<init> Opening Searcher@2401ffbc main
[junit4] 2> 927738 T3088 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 927738 T3088 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 927740 T3089 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2401ffbc main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 927741 T3088 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 927741 T3088 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:34548/ov_v collection:control_collection shard:shard1
[junit4] 2> 927741 T3088 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 927744 T3088 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 927748 T3088 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 927750 T3088 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 927750 T3088 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 927750 T3088 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:34548/ov_v/collection1/
[junit4] 2> 927750 T3088 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 927750 T3088 oasc.SyncStrategy.syncToMe http://127.0.0.1:34548/ov_v/collection1/ has no replicas
[junit4] 2> 927751 T3088 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:34548/ov_v/collection1/ shard1
[junit4] 2> 927751 T3088 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 928345 T3086 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 928349 T3085 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 928356 T3088 oasc.ZkController.register We are http://127.0.0.1:34548/ov_v/collection1/ and leader is http://127.0.0.1:34548/ov_v/collection1/
[junit4] 2> 928356 T3088 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:34548/ov_v
[junit4] 2> 928356 T3088 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 928356 T3088 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 928356 T3088 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 928357 T3088 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 928358 T3063 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 928358 T3063 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 928359 T3063 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 928361 T3063 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4] 2> 928362 T3063 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 928363 T3092 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5fe4fa2 name:ZooKeeperConnection Watcher:127.0.0.1:36230/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 928363 T3063 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 928364 T3063 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 928365 T3063 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4] 2> 928430 T3063 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 928432 T3063 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59191
[junit4] 2> 928433 T3063 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 928433 T3063 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 928433 T3063 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376392779590
[junit4] 2> 928434 T3063 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376392779590/'
[junit4] 2> 928443 T3063 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376392779590/solr.xml
[junit4] 2> 928479 T3063 oasc.CoreContainer.<init> New CoreContainer 1999633734
[junit4] 2> 928480 T3063 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376392779590/]
[junit4] 2> 928481 T3063 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 928481 T3063 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 928481 T3063 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 928482 T3063 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 928482 T3063 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 928482 T3063 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 928482 T3063 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 928482 T3063 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 928483 T3063 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 928483 T3063 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 928486 T3063 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36230/solr
[junit4] 2> 928486 T3063 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 928487 T3063 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 928488 T3103 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6cf1bdd9 name:ZooKeeperConnection Watcher:127.0.0.1:36230 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 928488 T3063 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 928489 T3063 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 928492 T3063 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 928493 T3105 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3baf56f2 name:ZooKeeperConnection Watcher:127.0.0.1:36230/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 928493 T3063 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 928495 T3063 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 929497 T3063 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59191_ov_v
[junit4] 2> 929498 T3063 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59191_ov_v
[junit4] 2> 929501 T3085 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 929501 T3105 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 929501 T3092 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 929501 T3085 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 929505 T3106 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 929506 T3106 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 929507 T3106 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 929855 T3086 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 929856 T3086 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:34548_ov_v",
[junit4] 2> "base_url":"http://127.0.0.1:34548/ov_v"}
[junit4] 2> 929860 T3086 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:59191_ov_v",
[junit4] 2> "base_url":"http://127.0.0.1:59191/ov_v"}
[junit4] 2> 929860 T3086 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 929860 T3086 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 929863 T3105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 929863 T3092 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 929863 T3085 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 930507 T3106 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 930507 T3106 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376392779590/collection1
[junit4] 2> 930507 T3106 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 930508 T3106 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 930508 T3106 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 930509 T3106 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376392779590/collection1/'
[junit4] 2> 930509 T3106 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376392779590/collection1/lib/README' to classloader
[junit4] 2> 930509 T3106 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376392779590/collection1/lib/classes/' to classloader
[junit4] 2> 930534 T3106 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 930563 T3106 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 930564 T3106 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 930570 T3106 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 930877 T3106 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 930882 T3106 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 930884 T3106 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 930892 T3106 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 930895 T3106 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 930897 T3106 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 930897 T3106 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 930897 T3106 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 930898 T3106 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 930898 T3106 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 930898 T3106 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 930898 T3106 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 930899 T3106 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376392779590/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/jetty1/
[junit4] 2> 930899 T3106 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7d9acbbd
[junit4] 2> 930899 T3106 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/jetty1
[junit4] 2> 930900 T3106 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/jetty1/index/
[junit4] 2> 930900 T3106 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 930900 T3106 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/jetty1/index
[junit4] 2> 930901 T3106 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@617e6614 lockFactory=org.apache.lucene.store.NativeFSLockFactory@52b0a9e0),segFN=segments_1,generation=1}
[junit4] 2> 930901 T3106 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 930904 T3106 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 930904 T3106 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 930905 T3106 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 930905 T3106 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 930906 T3106 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 930906 T3106 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 930906 T3106 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 930906 T3106 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 930907 T3106 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 930909 T3106 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 930911 T3106 oass.SolrIndexSearcher.<init> Opening Searcher@4e317904 main
[junit4] 2> 930911 T3106 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 930911 T3106 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 930915 T3107 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4e317904 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 930916 T3106 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 930916 T3106 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59191/ov_v collection:collection1 shard:shard1
[junit4] 2> 930916 T3106 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 930919 T3106 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 930923 T3106 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 930924 T3106 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 930924 T3106 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 930924 T3106 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59191/ov_v/collection1/
[junit4] 2> 930924 T3106 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 930925 T3106 oasc.SyncStrategy.syncToMe http://127.0.0.1:59191/ov_v/collection1/ has no replicas
[junit4] 2> 930925 T3106 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59191/ov_v/collection1/ shard1
[junit4] 2> 930925 T3106 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 931367 T3086 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 931372 T3092 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 931372 T3085 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 931372 T3105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 931379 T3106 oasc.ZkController.register We are http://127.0.0.1:59191/ov_v/collection1/ and leader is http://127.0.0.1:59191/ov_v/collection1/
[junit4] 2> 931380 T3106 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59191/ov_v
[junit4] 2> 931380 T3106 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 931380 T3106 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 931380 T3106 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 931382 T3106 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 931383 T3063 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 931383 T3063 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 931383 T3063 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 931446 T3063 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 931447 T3063 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42347
[junit4] 2> 931449 T3063 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 931449 T3063 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 931449 T3063 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376392782611
[junit4] 2> 931450 T3063 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376392782611/'
[junit4] 2> 931459 T3063 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376392782611/solr.xml
[junit4] 2> 931488 T3063 oasc.CoreContainer.<init> New CoreContainer 1334927866
[junit4] 2> 931489 T3063 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376392782611/]
[junit4] 2> 931489 T3063 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 931490 T3063 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 931490 T3063 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 931490 T3063 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 931490 T3063 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 931491 T3063 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 931491 T3063 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 931491 T3063 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 931491 T3063 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 931491 T3063 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 931494 T3063 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36230/solr
[junit4] 2> 931495 T3063 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 931495 T3063 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 931496 T3119 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25e53bb2 name:ZooKeeperConnection Watcher:127.0.0.1:36230 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 931497 T3063 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 931497 T3063 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 931500 T3063 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 931501 T3121 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d61931d name:ZooKeeperConnection Watcher:127.0.0.1:36230/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 931501 T3063 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 931504 T3063 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 932506 T3063 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42347_ov_v
[junit4] 2> 932507 T3063 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42347_ov_v
[junit4] 2> 932508 T3092 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 932508 T3121 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 932509 T3085 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 932508 T3105 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 932509 T3092 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 932509 T3105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 932509 T3085 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 932513 T3122 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 932513 T3122 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 932514 T3122 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 932876 T3086 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 932877 T3086 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:59191_ov_v",
[junit4] 2> "base_url":"http://127.0.0.1:59191/ov_v"}
[junit4] 2> 932881 T3086 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:42347_ov_v",
[junit4] 2> "base_url":"http://127.0.0.1:42347/ov_v"}
[junit4] 2> 932881 T3086 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 932881 T3086 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 932883 T3105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 932883 T3121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 932883 T3085 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 932883 T3092 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 933514 T3122 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 933515 T3122 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376392782611/collection1
[junit4] 2> 933515 T3122 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 933515 T3122 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 933516 T3122 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 933516 T3122 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376392782611/collection1/'
[junit4] 2> 933517 T3122 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376392782611/collection1/lib/README' to classloader
[junit4] 2> 933517 T3122 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376392782611/collection1/lib/classes/' to classloader
[junit4] 2> 933548 T3122 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 933577 T3122 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 933578 T3122 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 933584 T3122 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 933878 T3122 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 933883 T3122 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 933885 T3122 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 933893 T3122 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 933895 T3122 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 933897 T3122 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 933898 T3122 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 933898 T3122 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 933899 T3122 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 933899 T3122 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 933899 T3122 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 933900 T3122 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 933900 T3122 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376392782611/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/jetty2/
[junit4] 2> 933900 T3122 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7d9acbbd
[junit4] 2> 933901 T3122 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/jetty2
[junit4] 2> 933901 T3122 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/jetty2/index/
[junit4] 2> 933901 T3122 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 933901 T3122 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/jetty2/index
[junit4] 2> 933902 T3122 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4520e6f1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5e908de1),segFN=segments_1,generation=1}
[junit4] 2> 933902 T3122 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 933905 T3122 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 933906 T3122 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 933906 T3122 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 933907 T3122 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 933907 T3122 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 933907 T3122 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 933907 T3122 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 933908 T3122 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 933908 T3122 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 933910 T3122 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 933912 T3122 oass.SolrIndexSearcher.<init> Opening Searcher@36f3669d main
[junit4] 2> 933913 T3122 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 933913 T3122 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 933916 T3123 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@36f3669d main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 933917 T3122 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 933917 T3122 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42347/ov_v collection:collection1 shard:shard2
[junit4] 2> 933917 T3122 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 933920 T3122 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 933924 T3122 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 933925 T3122 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 933926 T3122 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 933926 T3122 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42347/ov_v/collection1/
[junit4] 2> 933926 T3122 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 933926 T3122 oasc.SyncStrategy.syncToMe http://127.0.0.1:42347/ov_v/collection1/ has no replicas
[junit4] 2> 933926 T3122 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42347/ov_v/collection1/ shard2
[junit4] 2> 933926 T3122 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 934388 T3086 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 934393 T3085 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 934393 T3092 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 934394 T3121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 934393 T3105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 934434 T3122 oasc.ZkController.register We are http://127.0.0.1:42347/ov_v/collection1/ and leader is http://127.0.0.1:42347/ov_v/collection1/
[junit4] 2> 934434 T3122 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42347/ov_v
[junit4] 2> 934434 T3122 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 934434 T3122 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 934434 T3122 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 934436 T3122 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 934437 T3063 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 934437 T3063 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 934437 T3063 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 934499 T3063 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 934502 T3063 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60930
[junit4] 2> 934502 T3063 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 934502 T3063 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 934503 T3063 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376392785665
[junit4] 2> 934503 T3063 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376392785665/'
[junit4] 2> 934512 T3063 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376392785665/solr.xml
[junit4] 2> 934541 T3063 oasc.CoreContainer.<init> New CoreContainer 1750956191
[junit4] 2> 934541 T3063 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376392785665/]
[junit4] 2> 934542 T3063 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 934543 T3063 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 934543 T3063 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 934543 T3063 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 934543 T3063 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 934543 T3063 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 934544 T3063 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 934544 T3063 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 934544 T3063 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 934544 T3063 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 934547 T3063 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36230/solr
[junit4] 2> 934547 T3063 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 934548 T3063 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 934550 T3135 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ccaefcf name:ZooKeeperConnection Watcher:127.0.0.1:36230 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 934550 T3063 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 934551 T3063 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 934553 T3063 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 934554 T3137 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b642f48 name:ZooKeeperConnection Watcher:127.0.0.1:36230/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 934555 T3063 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 934557 T3063 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 935559 T3063 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60930_ov_v
[junit4] 2> 935561 T3063 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60930_ov_v
[junit4] 2> 935562 T3092 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 935563 T3085 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 935563 T3105 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 935563 T3085 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 935563 T3121 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 935563 T3137 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 935563 T3105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 935564 T3092 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 935563 T3121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 935568 T3138 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 935568 T3138 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 935569 T3138 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 935897 T3086 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 935898 T3086 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard2",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:42347_ov_v",
[junit4] 2> "base_url":"http://127.0.0.1:42347/ov_v"}
[junit4] 2> 935901 T3086 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:60930_ov_v",
[junit4] 2> "base_url":"http://127.0.0.1:60930/ov_v"}
[junit4] 2> 935901 T3086 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 935901 T3086 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 935905 T3092 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 935905 T3121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 935905 T3137 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 935905 T3105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 935905 T3085 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 936570 T3138 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 936570 T3138 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376392785665/collection1
[junit4] 2> 936570 T3138 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 936571 T3138 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 936571 T3138 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 936572 T3138 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376392785665/collection1/'
[junit4] 2> 936572 T3138 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376392785665/collection1/lib/README' to classloader
[junit4] 2> 936572 T3138 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376392785665/collection1/lib/classes/' to classloader
[junit4] 2> 936597 T3138 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 936626 T3138 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 936627 T3138 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 936633 T3138 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 936925 T3138 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 936930 T3138 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 936932 T3138 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 936941 T3138 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 936944 T3138 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 936946 T3138 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 936946 T3138 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 936946 T3138 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 936947 T3138 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 936947 T3138 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 936947 T3138 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 936948 T3138 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 936948 T3138 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376392785665/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/jetty3/
[junit4] 2> 936948 T3138 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7d9acbbd
[junit4] 2> 936949 T3138 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/jetty3
[junit4] 2> 936949 T3138 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/jetty3/index/
[junit4] 2> 936949 T3138 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 936950 T3138 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376392776073/jetty3/index
[junit4] 2> 936951 T3138 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4faf5dfb lockFactory=org.apache.lucene.store.NativeFSLockFactory@696e2c92),segFN=segments_1,generation=1}
[junit4] 2> 936951 T3138 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 936955 T3138 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 936955 T3138 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 936956 T3138 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 936956 T3138 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 936957 T3138 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 936957 T3138 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 936957 T3138 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 936958 T3138 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 936958 T3138 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 936961 T3138 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 936964 T3138 oass.SolrIndexSearcher.<init> Opening Searcher@25e4f36e main
[junit4] 2> 936964 T3138 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 936964 T3138 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 936973 T3139 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@25e4f36e main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 936974 T3138 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 936975 T3138 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60930/ov_v collection:collection1 shard:shard1
[junit4] 2> 936975 T3138 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 936979 T3138 oasc.ZkController.register We are http://127.0.0.1:60930/ov_v/collection1/ and leader is http://127.0.0.1:59191/ov_v/collection1/
[junit4] 2> 936979 T3138 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60930/ov_v
[junit4] 2> 936980 T3138 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 936980 T3138 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C2307 name=collection1 org.apache.solr.core.SolrCore@5ddfa912 url=http://127.0.0.1:60930/ov_v/collection1 node=127.0.0.1:60930_ov_v C2307_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:60930_ov_v, base_url=http://127.0.0.1:60930/ov_v}
[junit4] 2> 936980 T3140 C2307 P60930 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 936980 T3138 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 936980 T3140 C2307 P60930 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 936981 T3140 C2307 P60930 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4] 2> 936981 T3140 C2307 P60930 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 936981 T3063 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 936981 T3063 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 936981 T3140 C2307 P60930 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 936982 T3063 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 936986 T3101 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 937045 T3063 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 937047 T3063 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33287
[junit4] 2> 937047 T3063 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 937047 T3063 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 937048 T3063 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376392788210
[junit4] 2> 937048 T3063 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376392788210/'
[junit4] 2> 937057 T3063 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376392788210/solr.xml
[junit4] 2> 937087 T3063 oasc.CoreContainer.<init> New CoreContainer 1887569806
[junit4] 2> 937088 T3063 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376392788210/]
[junit4] 2> 937088 T3063 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 937089 T3063 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 937089 T3063 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 937089 T3063 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 937089 T3063 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 937090 T3063 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 937090 T3063 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 937090 T3063 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 937090 T3063 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 937091 T3063 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 937094 T3063 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:36230/solr
[junit4] 2> 937094 T3063 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 937095 T3063 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 937096 T3152 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44aff348 name:ZooKeeperConnection Watcher:127.0.0.1:36230 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 937096 T3063 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 937097 T3063 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 937100 T3063 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 937101 T3154 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b72ae02 name:ZooKeeperConnection Watcher:127.0.0.1:36230/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 937101 T3063 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 937103 T3063 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 937409 T3086 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 937409 T3086 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"recovering",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:60930_ov_v",
[junit4] 2> "base_url":"http://127.0.0.1:60930/ov_v"}
[junit4] 2> 937412 T3105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 937412 T3154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 937413 T3121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 937412 T3092 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 937413 T3137 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 937412 T3085 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 937986 T3101 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 937986 T3101 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node3&state=recovering&nodeName=127.0.0.1:60930_ov_v&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4] 2> 938105 T3063 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33287_ov_v
[junit4] 2> 938106 T3063 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33287_ov_v
[junit4] 2> 938107 T3137 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 938107 T3085 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 938107 T3092 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 938107 T3085 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4] 2> 938107 T3105 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 938108 T3105 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4] 2> 938107 T3121 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 938108 T3137 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 938107 T3154 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 938108 T3092 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4] 2> 938108 T3121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_
[...truncated too long message...]
lockSize=73 maxBlockSize=236), 59grand53_s=Lucene41(blocksize=128), 73grand9_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 55grand42_s=Lucene41(blocksize=128), 37grand4_s=PostingsFormat(name=Asserting), 71grand0_s=Lucene41(blocksize=128), 57grand43_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 39grand7_s=PostingsFormat(name=Direct), 13child2_s=PostingsFormat(name=Asserting), 70grand58_s=Lucene41(blocksize=128), 14grand9_s=PostingsFormat(name=Asserting), 61grand3_s=Lucene41(blocksize=128), 50grand18_s=PostingsFormat(name=Asserting), 67grand15_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 71grand25_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 55grand21_s=PostingsFormat(name=Direct), 22child12_s=PostingsFormat(name=Direct), 30grand29_s=Lucene41(blocksize=128), 43grand17_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 74grand63_s=PostingsFormat(name=Asserting), 58grand50_s=PostingsFormat(name=Asserting), 53grand27_s=PostingsFormat(name=Direct), 17grand15_s=Lucene41(blocksize=128), 33grand6_s=Lucene41(blocksize=128), 75grand25_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 32grand16_s=Lucene41(blocksize=128), 24grand9_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 53grand38_s=PostingsFormat(name=Direct), 0parent_f1_s=PostingsFormat(name=Asserting), 34grand26_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 37child10_s=PostingsFormat(name=Direct), 68grand40_s=Lucene41(blocksize=128), 59grand0_s=PostingsFormat(name=Asserting), 63grand24_s=PostingsFormat(name=Direct), 52grand18_s=Lucene41(blocksize=128), 64grand7_s=PostingsFormat(name=Direct), 9grand2_s=PostingsFormat(name=Direct), 20grand5_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 23child6_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 29child18_s=PostingsFormat(name=Asserting), 72grand65_s=PostingsFormat(name=Asserting), 38child33_s=PostingsFormat(name=Direct), 40grand18_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 71grand19_s=Lucene41(blocksize=128), 65grand36_s=PostingsFormat(name=Asserting), 34child16_s=PostingsFormat(name=Asserting), 73grand40_s=PostingsFormat(name=Asserting), 64grand48_s=Lucene41(blocksize=128), 26child13_s=Lucene41(blocksize=128), 35grand34_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 65grand42_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 64grand53_s=PostingsFormat(name=Asserting), 39grand10_s=PostingsFormat(name=Direct), 18grand16_s=Lucene41(blocksize=128), 69grand44_s=PostingsFormat(name=Direct), 34grand20_s=PostingsFormat(name=Direct), 5grand0_s=PostingsFormat(name=Direct), 71grand14_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 7grand2_s=PostingsFormat(name=Direct), 75grand63_s=PostingsFormat(name=Direct), 65grand7_s=Lucene41(blocksize=128), 37grand23_s=PostingsFormat(name=Direct), 57grand54_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 31grand3_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 20child5_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 59grand36_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 35child9_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 34grand15_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 28child7_s=PostingsFormat(name=Direct), 33child32_s=PostingsFormat(name=Direct), 38child0_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 62grand1_s=PostingsFormat(name=Direct), 70grand6_s=PostingsFormat(name=Direct), 45grand21_s=Lucene41(blocksize=128), 58grand54_s=PostingsFormat(name=Asserting), 16grand11_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 37child27_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 29child24_s=PostingsFormat(name=Direct), 36grand14_s=PostingsFormat(name=Asserting), 41grand22_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 56grand22_s=PostingsFormat(name=Direct), 18child14_s=PostingsFormat(name=Asserting), 49grand30_s=PostingsFormat(name=Asserting), 52grand7_s=PostingsFormat(name=Asserting), 62grand55_s=PostingsFormat(name=Asserting), 38grand13_s=PostingsFormat(name=Direct), 65grand20_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 38child23_s=PostingsFormat(name=Asserting), 14child5_s=PostingsFormat(name=Asserting), 28grand20_s=PostingsFormat(name=Asserting), 39grand18_s=PostingsFormat(name=Direct), 26child20_s=Lucene41(blocksize=128), 59grand41_s=PostingsFormat(name=Direct), 31child23_s=PostingsFormat(name=Direct), 72grand55_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 22grand14_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 74grand14_s=Lucene41(blocksize=128), 54grand1_s=PostingsFormat(name=Asserting), 20grand12_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 15child5_s=PostingsFormat(name=Direct), 33grand14_s=PostingsFormat(name=Direct), 33child0_s=PostingsFormat(name=Asserting), 37child30_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 40grand22_s=PostingsFormat(name=Asserting), 31grand29_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 45grand5_s=Lucene41(blocksize=128), 24child19_s=Lucene41(blocksize=128), 58grand7_s=Lucene41(blocksize=128), 55grand36_s=PostingsFormat(name=Direct), 68grand35_s=PostingsFormat(name=Asserting), 9child0_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 52grand51_s=PostingsFormat(name=Direct), 61grand44_s=PostingsFormat(name=Direct), 68grand62_s=Lucene41(blocksize=128), 58grand24_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 58grand35_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 19child17_s=Lucene41(blocksize=128), 70grand64_s=PostingsFormat(name=Direct), 33child8_s=PostingsFormat(name=Asserting), 73grand47_s=PostingsFormat(name=Direct), 36child27_s=Lucene41(blocksize=128), 49grand39_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 28grand14_s=Pulsing41(freqCutoff=11 minBlockSize=73 maxBlockSize=236), 57grand33_s=Lucene41(blocksize=128), 12child7_s=PostingsFormat(name=Asserting), 51grand50_s=PostingsFormat(name=Direct), 32child3_s=Lucene41(blocksize=128), 35child34_s=PostingsFormat(name=Direct), 40grand16_s=PostingsFormat(name=Direct), 31grand19_s=Lucene41(blocksize=128), 51grand0_s=PostingsFormat(name=Asserting), 18child7_s=Lucene41(blocksize=128), 28child20_s=PostingsFormat(name=Asserting), 14child12_s=Lucene41(blocksize=128), 65grand50_s=PostingsFormat(name=Asserting)}, docValues:{timestamp=DocValuesFormat(name=SimpleText)}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=fr_CH, timezone=Europe/Vienna
[junit4] 2> NOTE: Linux 3.2.0-51-generic amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=8,threads=1,free=92833808,total=454033408
[junit4] 2> NOTE: All tests run in this JVM: [UUIDFieldTest, ChaosMonkeySafeLeaderTest, SuggesterWFSTTest, TestOmitPositions, TestQuerySenderListener, TestStressReorder, TestDistributedGrouping, CurrencyFieldOpenExchangeTest, TestRandomFaceting, TestNumberUtils, TestSolrCoreProperties, TestComponentsName, BasicZkTest, TestRecovery, TestQueryTypes, FieldAnalysisRequestHandlerTest, DefaultValueUpdateProcessorTest, NoCacheHeaderTest, HighlighterConfigTest, HighlighterTest, OpenExchangeRatesOrgProviderTest, BadIndexSchemaTest, TestBinaryField, StatsComponentTest, XmlUpdateRequestHandlerTest, PluginInfoTest, SimplePostToolTest, TestAddFieldRealTimeGet, TestSchemaVersionResource, ConvertedLegacyTest, SpellCheckComponentTest, BasicDistributedZk2Test, OutputWriterTest, NumericFieldsTest, TestUpdate, TestPartialUpdateDeduplication, CollectionsAPIDistributedZkTest, SortByFunctionTest, HdfsBasicDistributedZk2Test, TestSchemaNameResource, PathHierarchyTokenizerFactoryTest, DirectSolrSpellCheckerTest, RequestHandlersTest, TestSolr4Spatial, XsltUpdateRequestHandlerTest, SpellingQueryConverterTest, TestCopyFieldCollectionResource, TestArbitraryIndexDir, ZkSolrClientTest, TestFiltering, ClusterStateTest, TestStressLucene, TestMultiCoreConfBootstrap, TestCodecSupport, TestCollationField, DocValuesMultiTest, TestCSVLoader, TestAtomicUpdateErrorCases, SuggesterTSTTest, TestQuerySenderNoQuery, LeaderElectionIntegrationTest, ShardRoutingCustomTest, TestCloudManagedSchema, TestRangeQuery, TestLazyCores, NotRequiredUniqueKeyTest, SpellCheckCollatorTest, TestCSVResponseWriter, SignatureUpdateProcessorFactoryTest, TestDefaultSimilarityFactory, AlternateDirectoryTest, DebugComponentTest, TestUtils, RAMDirectoryFactoryTest, TestAnalyzedSuggestions, DirectUpdateHandlerTest, TestFastWriter, SyncSliceTest, OpenCloseCoreStressTest, OverseerTest, FullSolrCloudDistribCmdsTest]
[junit4] Completed on J1 in 127.22s, 1 test, 1 error <<< FAILURES!
[...truncated 471 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:389: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:369: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:450: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1230: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:873: There were test failures: 320 suites, 1360 tests, 1 error, 35 ignored (6 assumptions)
Total time: 44 minutes 3 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_25 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_25) - Build #
6995 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6995/
Java: 64bit/jdk1.7.0_25 -XX:-UseCompressedOops -XX:+UseG1GC
12 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([C28918D5690B6C09]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=929, name=qtp40158861-929 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=929, name=qtp40158861-929 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=997, name=HashSessionScavenger-35, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=997, name=HashSessionScavenger-35, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=998, name=qtp1592928399-998 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=998, name=qtp1592928399-998 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=966, name=qtp1388421313-966, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=966, name=qtp1388421313-966, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at __randomizedtesting.SeedInfo.seed([C28918D5690B6C09]:0)
at java.util.HashMap.newKeyIterator(HashMap.java:940)
at java.util.HashMap$KeySet.iterator(HashMap.java:974)
at java.util.HashSet.iterator(HashSet.java:170)
at sun.nio.ch.Util$2.iterator(Util.java:303)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=949, name=qtp1586215173-949, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=949, name=qtp1586215173-949, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=982, name=qtp550345555-982, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=982, name=qtp550345555-982, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=981, name=qtp550345555-981 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=981, name=qtp550345555-981 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=930, name=qtp40158861-930, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=930, name=qtp40158861-930, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=941, name=Thread-403, state=RUNNABLE, group=Overseer state updater.]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=941, name=Thread-403, state=RUNNABLE, group=Overseer state updater.]
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:249)
at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:278)
at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:130)
at java.lang.Thread.run(Thread.java:724)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
Error Message:
Captured an uncaught exception in thread: Thread[id=921, name=SessionTracker, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=921, name=SessionTracker, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.lang.Long.valueOf(Long.java:577)
at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:150)
FAILED: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([C28918D5690B6C09]:0)
Build Log:
[...truncated 10357 lines...]
[junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
[junit4] 2> 327745 T918 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ohc/
[junit4] 2> 327752 T918 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-FullSolrCloudDistribCmdsTest-1376380159771
[junit4] 2> 327753 T918 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 327754 T919 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 327854 T918 oasc.ZkTestServer.run start zk server on port:56112
[junit4] 2> 327855 T918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 327858 T925 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a46aae6 name:ZooKeeperConnection Watcher:127.0.0.1:56112 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 327859 T918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 327859 T918 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 327864 T918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 327866 T927 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14ed7f8b name:ZooKeeperConnection Watcher:127.0.0.1:56112/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 327866 T918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 327867 T918 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 327869 T918 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 327871 T918 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 327873 T918 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 327875 T918 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 327876 T918 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 327879 T918 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 327880 T918 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 327883 T918 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 327883 T918 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 327885 T918 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 327886 T918 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 327888 T918 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 327888 T918 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 327890 T918 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 327890 T918 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 327893 T918 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 327893 T918 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 327896 T918 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 327896 T918 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 327898 T918 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 327899 T918 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 327901 T918 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 327901 T918 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 327985 T918 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 327988 T918 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55240
[junit4] 2> 327989 T918 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 327989 T918 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 327989 T918 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376380159924
[junit4] 2> 327990 T918 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376380159924/'
[junit4] 2> 328001 T918 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376380159924/solr.xml
[junit4] 2> 328037 T918 oasc.CoreContainer.<init> New CoreContainer 1920519262
[junit4] 2> 328037 T918 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376380159924/]
[junit4] 2> 328039 T918 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 328039 T918 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 328039 T918 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 328039 T918 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 328040 T918 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 328040 T918 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 328040 T918 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 328041 T918 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 328041 T918 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 328041 T918 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 328044 T918 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56112/solr
[junit4] 2> 328045 T918 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 328045 T918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 328047 T938 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f65403d name:ZooKeeperConnection Watcher:127.0.0.1:56112 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 328047 T918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 328049 T918 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 328052 T918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 328053 T940 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@665f3029 name:ZooKeeperConnection Watcher:127.0.0.1:56112/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 328053 T918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 328055 T918 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 328057 T918 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 328060 T918 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 328061 T918 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55240_ohc
[junit4] 2> 328062 T918 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55240_ohc
[junit4] 2> 328065 T918 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 328068 T918 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 328070 T918 oasc.Overseer.start Overseer (id=90202450150948867-127.0.0.1:55240_ohc-n_0000000000) starting
[junit4] 2> 328073 T918 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 328075 T942 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 328076 T918 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 328078 T918 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 328079 T918 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 328082 T941 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 328085 T943 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 328085 T943 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 328086 T943 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 329585 T941 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 329587 T941 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:55240_ohc",
[junit4] 2> "base_url":"http://127.0.0.1:55240/ohc"}
[junit4] 2> 329587 T941 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 329587 T941 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 329593 T940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 330086 T943 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 330087 T943 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376380159924/collection1
[junit4] 2> 330087 T943 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 330088 T943 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 330088 T943 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 330089 T943 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376380159924/collection1/'
[junit4] 2> 330090 T943 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376380159924/collection1/lib/README' to classloader
[junit4] 2> 330090 T943 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376380159924/collection1/lib/classes/' to classloader
[junit4] 2> 330118 T943 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 330149 T943 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 330151 T943 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 330158 T943 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 330497 T943 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 330503 T943 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 330506 T943 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 330518 T943 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 330521 T943 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 330524 T943 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 330525 T943 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 330526 T943 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 330526 T943 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 330527 T943 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 330527 T943 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 330527 T943 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 330528 T943 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-controljetty-1376380159924/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/control/data/
[junit4] 2> 330528 T943 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bf88f8f
[junit4] 2> 330529 T943 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/control/data
[junit4] 2> 330529 T943 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/control/data/index/
[junit4] 2> 330529 T943 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 330529 T943 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/control/data/index
[junit4] 2> 330531 T943 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@683e74b6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5039a3de),segFN=segments_1,generation=1}
[junit4] 2> 330531 T943 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 330534 T943 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 330535 T943 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 330535 T943 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 330536 T943 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 330536 T943 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 330536 T943 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 330537 T943 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 330537 T943 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 330537 T943 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 330540 T943 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 330542 T943 oass.SolrIndexSearcher.<init> Opening Searcher@77b1fa04 main
[junit4] 2> 330543 T943 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 330543 T943 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 330546 T944 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@77b1fa04 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 330546 T943 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 330546 T943 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55240/ohc collection:control_collection shard:shard1
[junit4] 2> 330547 T943 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 330550 T943 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 330555 T943 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 330557 T943 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 330557 T943 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 330558 T943 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55240/ohc/collection1/
[junit4] 2> 330558 T943 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 330558 T943 oasc.SyncStrategy.syncToMe http://127.0.0.1:55240/ohc/collection1/ has no replicas
[junit4] 2> 330558 T943 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55240/ohc/collection1/ shard1
[junit4] 2> 330558 T943 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 331097 T941 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 331103 T940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 331114 T943 oasc.ZkController.register We are http://127.0.0.1:55240/ohc/collection1/ and leader is http://127.0.0.1:55240/ohc/collection1/
[junit4] 2> 331114 T943 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55240/ohc
[junit4] 2> 331115 T943 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 331115 T943 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 331115 T943 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 331116 T943 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 331117 T918 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 331118 T918 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 331118 T918 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 331121 T918 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4] 2> 331123 T918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 331124 T947 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c3f4c5c name:ZooKeeperConnection Watcher:127.0.0.1:56112/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 331124 T918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 331126 T918 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 331128 T918 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4] 2> 331204 T918 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 331206 T918 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33340
[junit4] 2> 331207 T918 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 331207 T918 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 331207 T918 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376380163146
[junit4] 2> 331208 T918 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376380163146/'
[junit4] 2> 331219 T918 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376380163146/solr.xml
[junit4] 2> 331253 T918 oasc.CoreContainer.<init> New CoreContainer 1932158439
[junit4] 2> 331253 T918 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376380163146/]
[junit4] 2> 331254 T918 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 331254 T918 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 331255 T918 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 331255 T918 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 331255 T918 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 331256 T918 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 331256 T918 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 331256 T918 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 331256 T918 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 331257 T918 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 331260 T918 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56112/solr
[junit4] 2> 331260 T918 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 331261 T918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 331262 T958 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a8da35e name:ZooKeeperConnection Watcher:127.0.0.1:56112 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 331262 T918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 331265 T918 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 331268 T918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 331269 T960 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20187971 name:ZooKeeperConnection Watcher:127.0.0.1:56112/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 331269 T918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 331274 T918 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 332277 T918 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33340_ohc
[junit4] 2> 332279 T918 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33340_ohc
[junit4] 2> 332283 T940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 332283 T947 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 332283 T960 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 332284 T940 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 332290 T961 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 332290 T961 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 332292 T961 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 332608 T941 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 332609 T941 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:55240_ohc",
[junit4] 2> "base_url":"http://127.0.0.1:55240/ohc"}
[junit4] 2> 332613 T941 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:33340_ohc",
[junit4] 2> "base_url":"http://127.0.0.1:33340/ohc"}
[junit4] 2> 332613 T941 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 332613 T941 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 332618 T960 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 332618 T940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 332618 T947 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 333292 T961 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 333292 T961 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376380163146/collection1
[junit4] 2> 333292 T961 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 333293 T961 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 333293 T961 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 333294 T961 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376380163146/collection1/'
[junit4] 2> 333295 T961 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376380163146/collection1/lib/README' to classloader
[junit4] 2> 333295 T961 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376380163146/collection1/lib/classes/' to classloader
[junit4] 2> 333323 T961 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 333355 T961 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 333356 T961 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 333363 T961 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 333696 T961 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 333702 T961 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 333704 T961 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 333718 T961 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 333721 T961 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 333724 T961 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 333725 T961 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 333726 T961 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 333726 T961 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 333727 T961 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 333727 T961 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 333727 T961 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 333727 T961 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty1-1376380163146/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/jetty1/
[junit4] 2> 333728 T961 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bf88f8f
[junit4] 2> 333728 T961 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/jetty1
[junit4] 2> 333729 T961 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/jetty1/index/
[junit4] 2> 333729 T961 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 333729 T961 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/jetty1/index
[junit4] 2> 333730 T961 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@65103be7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@51802d6b),segFN=segments_1,generation=1}
[junit4] 2> 333731 T961 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 333734 T961 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 333734 T961 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 333735 T961 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 333735 T961 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 333736 T961 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 333736 T961 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 333736 T961 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 333737 T961 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 333737 T961 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 333739 T961 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 333742 T961 oass.SolrIndexSearcher.<init> Opening Searcher@434be272 main
[junit4] 2> 333742 T961 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 333743 T961 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 333747 T962 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@434be272 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 333749 T961 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 333749 T961 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33340/ohc collection:collection1 shard:shard1
[junit4] 2> 333749 T961 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 333752 T961 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 333757 T961 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 333759 T961 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 333760 T961 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 333760 T961 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33340/ohc/collection1/
[junit4] 2> 333760 T961 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 333760 T961 oasc.SyncStrategy.syncToMe http://127.0.0.1:33340/ohc/collection1/ has no replicas
[junit4] 2> 333760 T961 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33340/ohc/collection1/ shard1
[junit4] 2> 333760 T961 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 334124 T941 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 334130 T940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 334130 T947 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 334130 T960 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 334167 T961 oasc.ZkController.register We are http://127.0.0.1:33340/ohc/collection1/ and leader is http://127.0.0.1:33340/ohc/collection1/
[junit4] 2> 334167 T961 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33340/ohc
[junit4] 2> 334167 T961 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 334167 T961 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 334167 T961 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 334169 T961 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 334170 T918 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 334170 T918 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 334171 T918 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 334238 T918 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 334240 T918 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52069
[junit4] 2> 334241 T918 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 334241 T918 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 334241 T918 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376380166191
[junit4] 2> 334242 T918 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376380166191/'
[junit4] 2> 334252 T918 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376380166191/solr.xml
[junit4] 2> 334285 T918 oasc.CoreContainer.<init> New CoreContainer 1442075458
[junit4] 2> 334285 T918 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376380166191/]
[junit4] 2> 334286 T918 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 334286 T918 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 334287 T918 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 334287 T918 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 334287 T918 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 334288 T918 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 334288 T918 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 334288 T918 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 334288 T918 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 334289 T918 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 334292 T918 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56112/solr
[junit4] 2> 334292 T918 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 334293 T918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 334294 T974 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@92b80c5 name:ZooKeeperConnection Watcher:127.0.0.1:56112 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 334294 T918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 334296 T918 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 334299 T918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 334300 T976 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@155782ac name:ZooKeeperConnection Watcher:127.0.0.1:56112/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 334300 T918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 334304 T918 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 335307 T918 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52069_ohc
[junit4] 2> 335308 T918 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52069_ohc
[junit4] 2> 335310 T940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 335311 T947 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 335312 T947 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 335312 T976 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 335312 T960 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 335312 T960 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 335313 T940 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 335318 T977 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 335318 T977 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 335320 T977 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 335635 T941 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 335636 T941 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:33340_ohc",
[junit4] 2> "base_url":"http://127.0.0.1:33340/ohc"}
[junit4] 2> 335640 T941 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:52069_ohc",
[junit4] 2> "base_url":"http://127.0.0.1:52069/ohc"}
[junit4] 2> 335640 T941 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 335640 T941 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 335643 T960 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 335643 T940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 335643 T947 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 335643 T976 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 336320 T977 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 336320 T977 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376380166191/collection1
[junit4] 2> 336320 T977 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 336321 T977 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 336321 T977 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 336322 T977 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376380166191/collection1/'
[junit4] 2> 336323 T977 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376380166191/collection1/lib/README' to classloader
[junit4] 2> 336323 T977 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376380166191/collection1/lib/classes/' to classloader
[junit4] 2> 336351 T977 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 336382 T977 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 336383 T977 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 336391 T977 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 336730 T977 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 336736 T977 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 336738 T977 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 336752 T977 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 336756 T977 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 336759 T977 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 336760 T977 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 336760 T977 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 336760 T977 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 336761 T977 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 336761 T977 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 336762 T977 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 336762 T977 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty2-1376380166191/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/jetty2/
[junit4] 2> 336762 T977 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bf88f8f
[junit4] 2> 336763 T977 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/jetty2
[junit4] 2> 336763 T977 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/jetty2/index/
[junit4] 2> 336763 T977 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 336764 T977 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/jetty2/index
[junit4] 2> 336765 T977 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@63af8217 lockFactory=org.apache.lucene.store.NativeFSLockFactory@54f06699),segFN=segments_1,generation=1}
[junit4] 2> 336765 T977 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 336769 T977 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 336769 T977 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 336769 T977 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 336770 T977 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 336770 T977 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 336770 T977 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 336771 T977 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 336771 T977 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 336772 T977 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 336774 T977 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 336776 T977 oass.SolrIndexSearcher.<init> Opening Searcher@291b067d main
[junit4] 2> 336777 T977 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 336777 T977 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 336780 T978 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@291b067d main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 336782 T977 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 336782 T977 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52069/ohc collection:collection1 shard:shard2
[junit4] 2> 336782 T977 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 336786 T977 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 336791 T977 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 336793 T977 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 336793 T977 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 336794 T977 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52069/ohc/collection1/
[junit4] 2> 336794 T977 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 336794 T977 oasc.SyncStrategy.syncToMe http://127.0.0.1:52069/ohc/collection1/ has no replicas
[junit4] 2> 336794 T977 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52069/ohc/collection1/ shard2
[junit4] 2> 336794 T977 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 337149 T941 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 337157 T976 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 337157 T940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 337157 T960 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 337157 T947 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 337200 T977 oasc.ZkController.register We are http://127.0.0.1:52069/ohc/collection1/ and leader is http://127.0.0.1:52069/ohc/collection1/
[junit4] 2> 337200 T977 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52069/ohc
[junit4] 2> 337200 T977 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 337200 T977 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 337200 T977 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 337202 T977 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 337203 T918 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 337204 T918 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 337204 T918 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 337270 T918 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 337272 T918 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41462
[junit4] 2> 337273 T918 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 337274 T918 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 337274 T918 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376380169224
[junit4] 2> 337274 T918 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376380169224/'
[junit4] 2> 337285 T918 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376380169224/solr.xml
[junit4] 2> 337333 T918 oasc.CoreContainer.<init> New CoreContainer 750636506
[junit4] 2> 337334 T918 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376380169224/]
[junit4] 2> 337335 T918 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 337335 T918 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 337336 T918 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 337336 T918 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 337336 T918 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 337336 T918 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 337337 T918 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 337337 T918 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 337337 T918 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 337337 T918 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 337340 T918 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56112/solr
[junit4] 2> 337341 T918 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 337342 T918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 337343 T990 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7bd683f4 name:ZooKeeperConnection Watcher:127.0.0.1:56112 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 337343 T918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 337346 T918 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 337349 T918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 337350 T992 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15e3c47b name:ZooKeeperConnection Watcher:127.0.0.1:56112/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 337350 T918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 337354 T918 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 338357 T918 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41462_ohc
[junit4] 2> 338358 T918 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41462_ohc
[junit4] 2> 338360 T976 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 338361 T940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 338361 T947 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 338361 T992 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 338361 T960 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 338361 T947 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 338362 T960 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 338362 T976 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 338363 T940 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 338368 T993 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 338368 T993 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 338369 T993 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 338663 T941 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 338664 T941 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard2",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:52069_ohc",
[junit4] 2> "base_url":"http://127.0.0.1:52069/ohc"}
[junit4] 2> 338667 T941 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:41462_ohc",
[junit4] 2> "base_url":"http://127.0.0.1:41462/ohc"}
[junit4] 2> 338667 T941 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 338667 T941 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 338670 T992 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 338670 T960 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 338670 T947 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 338670 T940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 338670 T976 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 339370 T993 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 339370 T993 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376380169224/collection1
[junit4] 2> 339370 T993 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 339371 T993 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 339371 T993 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 339372 T993 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376380169224/collection1/'
[junit4] 2> 339372 T993 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376380169224/collection1/lib/README' to classloader
[junit4] 2> 339373 T993 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376380169224/collection1/lib/classes/' to classloader
[junit4] 2> 339402 T993 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 339434 T993 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 339436 T993 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 339443 T993 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 339789 T993 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 339795 T993 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 339797 T993 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 339812 T993 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 339815 T993 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 339818 T993 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 339819 T993 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 339819 T993 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 339820 T993 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 339821 T993 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 339821 T993 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 339821 T993 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 339821 T993 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty3-1376380169224/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/jetty3/
[junit4] 2> 339822 T993 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2bf88f8f
[junit4] 2> 339822 T993 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/jetty3
[junit4] 2> 339822 T993 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/jetty3/index/
[junit4] 2> 339823 T993 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 339823 T993 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1376380159771/jetty3/index
[junit4] 2> 339824 T993 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@171adcc0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@cb6beb),segFN=segments_1,generation=1}
[junit4] 2> 339825 T993 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 339828 T993 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 339828 T993 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 339829 T993 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 339829 T993 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 339830 T993 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 339830 T993 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 339830 T993 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 339831 T993 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 339831 T993 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 339833 T993 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 339836 T993 oass.SolrIndexSearcher.<init> Opening Searcher@31ce8bd5 main
[junit4] 2> 339836 T993 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 339836 T993 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 339840 T994 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@31ce8bd5 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 339842 T993 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 339842 T993 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41462/ohc collection:collection1 shard:shard1
[junit4] 2> 339842 T993 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
[junit4] 2> 339847 T993 oasc.ZkController.register We are http://127.0.0.1:41462/ohc/collection1/ and leader is http://127.0.0.1:33340/ohc/collection1/
[junit4] 2> 339848 T993 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41462/ohc
[junit4] 2> 339848 T993 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 339848 T993 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C1350 name=collection1 org.apache.solr.core.SolrCore@6bc93fad url=http://127.0.0.1:41462/ohc/collection1 node=127.0.0.1:41462_ohc C1350_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:41462_ohc, base_url=http://127.0.0.1:41462/ohc}
[junit4] 2> 339848 T995 C1350 P41462 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 339849 T993 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 339849 T995 C1350 P41462 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 339849 T995 C1350 P41462 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4] 2> 339850 T995 C1350 P41462 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 339850 T918 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 339850 T918 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 339851 T995 C1350 P41462 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 339851 T918 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 339856 T954 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 339924 T918 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 339927 T918 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59434
[junit4] 2> 339928 T918 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 339928 T918 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 339929 T918 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376380171873
[junit4] 2> 339929 T918 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376380171873/'
[junit4] 2> 339940 T918 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376380171873/solr.xml
[junit4] 2> 339974 T918 oasc.CoreContainer.<init> New CoreContainer 2030357351
[junit4] 2> 339975 T918 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-jetty4-1376380171873/]
[junit4] 2> 339976 T918 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 339976 T918 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 339976 T918 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 339977 T918 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 339977 T918 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 339977 T918 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 339978 T918 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 339978 T918 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 339978 T918 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 339978 T918 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 339981 T918 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56112/solr
[junit4] 2> 339982 T918 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 339983 T918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 339984 T1007 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2fcbdf23 name:ZooKeeperConnection Watcher:127.0.0.1:56112 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 339984 T918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 339986 T918 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 339989 T918 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 339991 T1009 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7fd3bfa8 name:ZooKeeperConnection Watcher:127.0.0.1:56112/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 339991 T918 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 339995 T918 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 340176 T941 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 340177 T941 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"recovering",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:41462_ohc",
[junit4] 2> "base_url":"http://127.0.0.1:41462/ohc"}
[junit4] 2> 340415 T976 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 340415 T1009 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 340415 T992 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 340415 T947 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 340415 T940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 340415 T960 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 340856 T954 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 340857 T954 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node3&state=recovering&nodeName=127.0.0.1:41462_ohc&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4] 2> 340998 T918 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59434_ohc
[junit4] 2> 341000 T918 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59434_ohc
[junit4] 2> 341002 T992 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 341002 T940 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurre
[...truncated too long message...]
a.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4] 2> 23) Thread[id=978, name=searcherExecutor-568-thread-1, state=WAITING, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4] 2> 24) Thread[id=965, name=qtp1388421313-965 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
[junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
[junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
[junit4] 2> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4] 2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=ja, timezone=America/New_York
[junit4] 2> NOTE: Linux 3.2.0-51-generic amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=8,threads=28,free=67591112,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [TestIBSimilarityFactory, AssignTest, TestWordDelimiterFilterFactory, TestIndexingPerformance, ChaosMonkeySafeLeaderTest, TestSchemaVersionResource, TestXIncludeConfig, QueryResultKeyTest, ShardRoutingCustomTest, SortByFunctionTest, SolrPluginUtilsTest, HardAutoCommitTest, StandardRequestHandlerTest, TestCSVLoader, SolrRequestParserTest, OpenExchangeRatesOrgProviderTest, TestSchemaNameResource, HdfsBasicDistributedZk2Test, TestManagedSchemaFieldResource, TestJmxIntegration, TestSuggestSpellingConverter, XsltUpdateRequestHandlerTest, PingRequestHandlerTest, TestCoreDiscovery, DocValuesMultiTest, TestSolrQueryParserResource, TestSweetSpotSimilarityFactory, DateMathParserTest, HighlighterConfigTest, NoCacheHeaderTest, TestQuerySenderNoQuery, RequestHandlersTest, IndexSchemaRuntimeFieldTest, TestMultiCoreConfBootstrap, StatsComponentTest, TestStressRecovery, TestInfoStreamLogging, SpellingQueryConverterTest, RequiredFieldsTest, TestSolrDeletionPolicy2, HdfsSyncSliceTest, TestReloadAndDeleteDocs, TestArbitraryIndexDir, BadCopyFieldTest, DeleteShardTest, NumericFieldsTest, TestBinaryField, TestRangeQuery, TestHashPartitioner, TestCollationField, TestRandomMergePolicy, PluginInfoTest, ClusterStateTest, FullSolrCloudDistribCmdsTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=FullSolrCloudDistribCmdsTest -Dtests.seed=C28918D5690B6C09 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ja -Dtests.timezone=America/New_York -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J1 | FullSolrCloudDistribCmdsTest (suite) <<<
[junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4] > at __randomizedtesting.SeedInfo.seed([C28918D5690B6C09]:0)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=929, name=qtp40158861-929 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=997, name=HashSessionScavenger-35, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=998, name=qtp1592928399-998 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=966, name=qtp1388421313-966, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space
[junit4] > at __randomizedtesting.SeedInfo.seed([C28918D5690B6C09]:0)
[junit4] > at java.util.HashMap.newKeyIterator(HashMap.java:940)
[junit4] > at java.util.HashMap$KeySet.iterator(HashMap.java:974)
[junit4] > at java.util.HashSet.iterator(HashSet.java:170)
[junit4] > at sun.nio.ch.Util$2.iterator(Util.java:303)
[junit4] > at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
[junit4] > at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] > at java.lang.Thread.run(Thread.java:724)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=949, name=qtp1586215173-949, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=982, name=qtp550345555-982, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #8: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=981, name=qtp550345555-981 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #9: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=930, name=qtp40158861-930, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #10: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=941, name=Thread-403, state=RUNNABLE, group=Overseer state updater.]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:249)
[junit4] > at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:278)
[junit4] > at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:130)
[junit4] > at java.lang.Thread.run(Thread.java:724)Throwable #11: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=921, name=SessionTracker, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4] > Caused by: java.lang.OutOfMemoryError: Java heap space
[junit4] > at java.lang.Long.valueOf(Long.java:577)
[junit4] > at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:150)
[junit4] Completed on J1 in 7223.56s, 1 test, 12 errors <<< FAILURES!
[...truncated 106 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:389: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:369: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:450: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1230: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:873: There were test failures: 320 suites, 1360 tests, 11 suite-level errors, 1 error, 128 ignored (5 assumptions)
Total time: 145 minutes 42 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_25 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure