You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/30 01:54:07 UTC
[JENKINS] Lucene-Solr-4.x-Linux (32bit/ibm-j9-jdk6) - Build # 6297
- Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6297/
Java: 32bit/ibm-j9-jdk6 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
1 tests failed.
FAILED: org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch
Error Message:
Server at http://127.0.0.1:40848/onenodecollectioncore returned non ok status:404, message:Can not find: /onenodecollectioncore/update
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:40848/onenodecollectioncore returned non ok status:404, message:Can not find: /onenodecollectioncore/update
at __randomizedtesting.SeedInfo.seed([CA85B2609398554B:4B633C78E4C73577]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:198)
at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:90)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:738)
Build Log:
[...truncated 8652 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4:junit4] 2> log4j:WARN No such property [conversionPattern] in org.apache.solr.util.SolrLogLayout.
[junit4:junit4] 2> 436 T18 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 461 T18 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372548599286
[junit4:junit4] 2> 467 T18 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 470 T19 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 681 T18 oasc.ZkTestServer.run start zk server on port:42922
[junit4:junit4] 2> 730 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 780 T27 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@280a280a name:ZooKeeperConnection Watcher:127.0.0.1:42922 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 781 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 783 T18 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 808 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 819 T29 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b681b68 name:ZooKeeperConnection Watcher:127.0.0.1:42922/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 824 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 830 T18 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 843 T18 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 857 T18 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 871 T18 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 887 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 906 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 930 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 932 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 946 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 948 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 966 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 968 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 975 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 977 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 984 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 986 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 995 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 996 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 1016 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 1018 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 1030 T18 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 1032 T18 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 1368 T18 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 1468 T18 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50832
[junit4:junit4] 2> 1481 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 1509 T18 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 1511 T18 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372548599868
[junit4:junit4] 2> 1520 T18 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372548599868/solr.xml
[junit4:junit4] 2> 1527 T18 oasc.CoreContainer.<init> New CoreContainer 1569807761
[junit4:junit4] 2> 1529 T18 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372548599868/'
[junit4:junit4] 2> 1531 T18 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372548599868/'
[junit4:junit4] 2> 1897 T18 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 1898 T18 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 1900 T18 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 1901 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 1902 T18 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 1903 T18 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 1904 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 1905 T18 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 1906 T18 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 1914 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 2313 T18 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 2316 T18 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42922/solr
[junit4:junit4] 2> 2322 T18 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 2325 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2332 T42 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41884188 name:ZooKeeperConnection Watcher:127.0.0.1:42922 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2333 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2341 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 2363 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 2367 T44 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16771677 name:ZooKeeperConnection Watcher:127.0.0.1:42922/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 2376 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 2383 T18 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 2397 T18 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 2414 T18 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 2422 T18 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50832_
[junit4:junit4] 2> 2427 T18 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50832_
[junit4:junit4] 2> 2477 T18 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 2499 T18 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 2506 T18 oasc.Overseer.start Overseer (id=89951345013161987-127.0.0.1:50832_-n_0000000000) starting
[junit4:junit4] 2> 2518 T18 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 2536 T46 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 2537 T18 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 2550 T18 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 2556 T18 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 2569 T45 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 2597 T47 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 2598 T47 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 4094 T45 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 4097 T45 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50832",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50832_",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 4099 T45 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 4105 T45 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 4119 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 4602 T47 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372548599868/collection1
[junit4:junit4] 2> 4603 T47 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 4607 T47 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 4608 T47 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 4614 T47 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372548599868/collection1/'
[junit4:junit4] 2> 4618 T47 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372548599868/collection1/lib/README' to classloader
[junit4:junit4] 2> 4620 T47 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372548599868/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 4799 T47 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 4947 T47 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 4956 T47 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 5105 T47 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 6018 T47 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 6019 T47 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 6028 T47 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 6048 T47 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 6052 T47 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 6181 T47 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 6190 T47 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 6197 T47 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 6202 T47 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 6202 T47 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 6204 T47 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 6207 T47 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 6208 T47 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 6220 T47 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 6225 T47 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1372548599868/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/control/data/
[junit4:junit4] 2> 6260 T47 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56ac56ac
[junit4:junit4] 2> 6298 T47 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 6314 T47 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/control/data
[junit4:junit4] 2> 6316 T47 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/control/data/index/
[junit4:junit4] 2> 6317 T47 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 6321 T47 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/control/data/index
[junit4:junit4] 2> 6374 T47 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@32c232c2 lockFactory=org.apache.lucene.store.NativeFSLockFactory@60d460d4),segFN=segments_1,generation=1}
[junit4:junit4] 2> 6376 T47 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 6475 T47 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 6476 T47 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 6487 T47 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 6495 T47 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 6498 T47 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 6499 T47 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 6501 T47 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 6506 T47 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 6508 T47 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 6526 T47 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 6540 T47 oass.SolrIndexSearcher.<init> Opening Searcher@6d636d63 main
[junit4:junit4] 2> 6557 T47 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/control/data/tlog
[junit4:junit4] 2> 6562 T47 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 6563 T47 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 6600 T48 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6d636d63 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 6609 T47 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 6610 T47 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50832 collection:control_collection shard:shard1
[junit4:junit4] 2> 6613 T47 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 6626 T47 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 6645 T47 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 6653 T47 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 6654 T47 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 6655 T47 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50832/collection1/
[junit4:junit4] 2> 6656 T47 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 6657 T47 oasc.SyncStrategy.syncToMe http://127.0.0.1:50832/collection1/ has no replicas
[junit4:junit4] 2> 6658 T47 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50832/collection1/
[junit4:junit4] 2> 6658 T47 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 7135 T45 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 7252 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 7283 T47 oasc.ZkController.register We are http://127.0.0.1:50832/collection1/ and leader is http://127.0.0.1:50832/collection1/
[junit4:junit4] 2> 7285 T47 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50832
[junit4:junit4] 2> 7286 T47 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 7287 T47 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 7288 T47 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 7293 T47 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 7298 T18 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 7300 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 7310 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 7327 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 7331 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 7334 T51 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e990e99 name:ZooKeeperConnection Watcher:127.0.0.1:42922/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 7336 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 7339 T18 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 7386 T18 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 7591 T18 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 7596 T18 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50340
[junit4:junit4] 2> 7598 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 7600 T18 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 7601 T18 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372548606209
[junit4:junit4] 2> 7603 T18 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372548606209/solr.xml
[junit4:junit4] 2> 7613 T18 oasc.CoreContainer.<init> New CoreContainer 1899917630
[junit4:junit4] 2> 7615 T18 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372548606209/'
[junit4:junit4] 2> 7617 T18 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372548606209/'
[junit4:junit4] 2> 7720 T18 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 7722 T18 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 7723 T18 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 7724 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 7726 T18 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 7727 T18 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 7728 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 7730 T18 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 7731 T18 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 7733 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 7748 T18 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 7750 T18 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42922/solr
[junit4:junit4] 2> 7751 T18 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 7753 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 7756 T62 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@468d468d name:ZooKeeperConnection Watcher:127.0.0.1:42922 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 7758 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 7762 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 7778 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 7785 T64 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44984498 name:ZooKeeperConnection Watcher:127.0.0.1:42922/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 7787 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 7804 T18 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 8772 T45 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 8775 T45 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50832",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:50832__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:50832_",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 8791 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 8791 T51 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 8795 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 8837 T18 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50340_
[junit4:junit4] 2> 8839 T18 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50340_
[junit4:junit4] 2> 8847 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 8849 T51 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 8850 T51 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 8852 T64 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 8853 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 8858 T44 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 8934 T65 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 8935 T65 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 10312 T45 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 10317 T45 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50340",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50340_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 10319 T45 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 10320 T45 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 10431 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 10431 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 10438 T51 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 10939 T65 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372548606209/collection1
[junit4:junit4] 2> 10941 T65 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 10945 T65 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 10946 T65 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 10953 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372548606209/collection1/'
[junit4:junit4] 2> 10956 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372548606209/collection1/lib/README' to classloader
[junit4:junit4] 2> 10958 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372548606209/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 11052 T65 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 11144 T65 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 11153 T65 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 11179 T65 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 11941 T65 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 11942 T65 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 11944 T65 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 11961 T65 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 11967 T65 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 12027 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 12035 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 12044 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 12055 T65 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 12056 T65 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 12057 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 12059 T65 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 12060 T65 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 12061 T65 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 12062 T65 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372548606209/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty1/
[junit4:junit4] 2> 12063 T65 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56ac56ac
[junit4:junit4] 2> 12064 T65 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 12066 T65 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty1
[junit4:junit4] 2> 12066 T65 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty1/index/
[junit4:junit4] 2> 12067 T65 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 12068 T65 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty1/index
[junit4:junit4] 2> 12075 T65 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2cba2cba lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d765d76),segFN=segments_1,generation=1}
[junit4:junit4] 2> 12076 T65 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 12082 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 12083 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 12085 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 12087 T65 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 12089 T65 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 12090 T65 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 12091 T65 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 12093 T65 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 12094 T65 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 12096 T65 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 12099 T65 oass.SolrIndexSearcher.<init> Opening Searcher@31203120 main
[junit4:junit4] 2> 12100 T65 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty1/tlog
[junit4:junit4] 2> 12102 T65 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 12103 T65 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 12120 T66 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@31203120 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 12130 T65 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 12132 T65 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50340 collection:collection1 shard:shard1
[junit4:junit4] 2> 12137 T65 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 12179 T65 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 12188 T65 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 12189 T65 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 12189 T65 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50340/collection1/
[junit4:junit4] 2> 12190 T65 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 12191 T65 oasc.SyncStrategy.syncToMe http://127.0.0.1:50340/collection1/ has no replicas
[junit4:junit4] 2> 12192 T65 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50340/collection1/
[junit4:junit4] 2> 12194 T65 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 13454 T45 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 13477 T51 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 13477 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 13477 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 13486 T65 oasc.ZkController.register We are http://127.0.0.1:50340/collection1/ and leader is http://127.0.0.1:50340/collection1/
[junit4:junit4] 2> 13487 T65 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50340
[junit4:junit4] 2> 13488 T65 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 13489 T65 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 13490 T65 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 13494 T65 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 13498 T18 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 13499 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 13501 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 13816 T18 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 13819 T18 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40848
[junit4:junit4] 2> 13821 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 13821 T18 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 13822 T18 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372548612335
[junit4:junit4] 2> 13823 T18 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372548612335/solr.xml
[junit4:junit4] 2> 13824 T18 oasc.CoreContainer.<init> New CoreContainer 646129283
[junit4:junit4] 2> 13825 T18 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372548612335/'
[junit4:junit4] 2> 13826 T18 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372548612335/'
[junit4:junit4] 2> 13912 T18 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 13913 T18 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 13913 T18 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 13914 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 13915 T18 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 13916 T18 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 13918 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 13919 T18 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 13920 T18 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 13922 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 13935 T18 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 13938 T18 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42922/solr
[junit4:junit4] 2> 13939 T18 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 13941 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 13946 T78 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@511d511d name:ZooKeeperConnection Watcher:127.0.0.1:42922 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 13947 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 13952 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 13978 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 13989 T80 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b634b63 name:ZooKeeperConnection Watcher:127.0.0.1:42922/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 13990 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 14002 T18 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 15000 T45 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 15003 T45 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50340",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:50340__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:50340_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 15012 T18 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40848_
[junit4:junit4] 2> 15015 T18 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40848_
[junit4:junit4] 2> 15022 T51 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 15022 T64 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 15022 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 15024 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15023 T51 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15024 T80 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 15027 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15028 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15030 T51 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15093 T81 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 15094 T81 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 15095 T44 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 15096 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 16544 T45 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 16547 T45 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40848",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40848_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 16548 T45 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 16549 T45 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 16563 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 16563 T51 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 16563 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 16563 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 17097 T81 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372548612335/collection1
[junit4:junit4] 2> 17098 T81 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 17102 T81 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 17103 T81 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 17106 T81 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372548612335/collection1/'
[junit4:junit4] 2> 17109 T81 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372548612335/collection1/lib/README' to classloader
[junit4:junit4] 2> 17110 T81 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372548612335/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 17158 T81 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 17214 T81 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 17218 T81 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 17228 T81 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 17711 T81 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 17712 T81 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 17713 T81 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 17723 T81 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 17727 T81 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 17777 T81 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 17784 T81 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 17791 T81 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 17794 T81 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 17794 T81 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 17795 T81 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 17797 T81 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 17798 T81 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 17799 T81 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 17799 T81 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1372548612335/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty2/
[junit4:junit4] 2> 17800 T81 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56ac56ac
[junit4:junit4] 2> 17802 T81 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 17803 T81 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty2
[junit4:junit4] 2> 17804 T81 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty2/index/
[junit4:junit4] 2> 17804 T81 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 17805 T81 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty2/index
[junit4:junit4] 2> 17810 T81 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@21622162 lockFactory=org.apache.lucene.store.NativeFSLockFactory@17bd17bd),segFN=segments_1,generation=1}
[junit4:junit4] 2> 17811 T81 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 17818 T81 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 17819 T81 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 17821 T81 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 17822 T81 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 17824 T81 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 17825 T81 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 17826 T81 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 17827 T81 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 17828 T81 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 17831 T81 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 17834 T81 oass.SolrIndexSearcher.<init> Opening Searcher@1e821e82 main
[junit4:junit4] 2> 17835 T81 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty2/tlog
[junit4:junit4] 2> 17837 T81 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 17837 T81 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 17857 T82 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e821e82 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 17862 T81 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 17863 T81 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40848 collection:collection1 shard:shard2
[junit4:junit4] 2> 17867 T81 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 17885 T81 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 17902 T81 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 17902 T81 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 17903 T81 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40848/collection1/
[junit4:junit4] 2> 17904 T81 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 17904 T81 oasc.SyncStrategy.syncToMe http://127.0.0.1:40848/collection1/ has no replicas
[junit4:junit4] 2> 17905 T81 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40848/collection1/
[junit4:junit4] 2> 17906 T81 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 18075 T45 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 18100 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 18101 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 18101 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 18101 T51 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 18118 T81 oasc.ZkController.register We are http://127.0.0.1:40848/collection1/ and leader is http://127.0.0.1:40848/collection1/
[junit4:junit4] 2> 18119 T81 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40848
[junit4:junit4] 2> 18119 T81 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 18120 T81 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 18121 T81 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 18124 T81 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 18127 T18 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 18128 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 18129 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 18354 T18 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 18357 T18 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51603
[junit4:junit4] 2> 18359 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 18360 T18 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 18361 T18 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372548616960
[junit4:junit4] 2> 18361 T18 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372548616960/solr.xml
[junit4:junit4] 2> 18362 T18 oasc.CoreContainer.<init> New CoreContainer 900412843
[junit4:junit4] 2> 18363 T18 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372548616960/'
[junit4:junit4] 2> 18364 T18 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372548616960/'
[junit4:junit4] 2> 18453 T18 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 18454 T18 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 18455 T18 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 18457 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 18458 T18 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 18459 T18 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 18460 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 18461 T18 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 18462 T18 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 18464 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 18479 T18 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 18482 T18 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42922/solr
[junit4:junit4] 2> 18483 T18 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 18486 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 18489 T94 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@615d615d name:ZooKeeperConnection Watcher:127.0.0.1:42922 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 18490 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 18495 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 18508 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 18511 T96 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@53ca53ca name:ZooKeeperConnection Watcher:127.0.0.1:42922/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 18512 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 18530 T18 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 19538 T18 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51603_
[junit4:junit4] 2> 19540 T18 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51603_
[junit4:junit4] 2> 19544 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 19546 T64 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 19547 T80 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 19548 T96 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 19549 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 19551 T51 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 19552 T51 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 19547 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 19554 T44 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 19564 T97 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 19565 T97 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 19621 T45 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 19623 T45 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40848",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:40848__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:40848_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 19633 T45 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51603",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:51603_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 19634 T45 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 19635 T45 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 19647 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 19647 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 19648 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 19647 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 19647 T51 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 20568 T97 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372548616960/collection1
[junit4:junit4] 2> 20569 T97 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 20571 T97 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 20572 T97 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 20576 T97 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372548616960/collection1/'
[junit4:junit4] 2> 20579 T97 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372548616960/collection1/lib/README' to classloader
[junit4:junit4] 2> 20581 T97 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372548616960/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 20633 T97 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 20707 T97 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 20712 T97 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 20725 T97 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 21372 T97 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 21373 T97 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 21374 T97 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 21384 T97 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 21388 T97 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 21450 T97 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 21460 T97 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 21471 T97 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 21476 T97 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 21477 T97 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 21479 T97 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 21484 T97 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 21485 T97 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 21487 T97 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 21488 T97 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1372548616960/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty3/
[junit4:junit4] 2> 21489 T97 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56ac56ac
[junit4:junit4] 2> 21492 T97 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 21494 T97 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty3
[junit4:junit4] 2> 21495 T97 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty3/index/
[junit4:junit4] 2> 21496 T97 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 21498 T97 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty3/index
[junit4:junit4] 2> 21504 T97 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@190c190c lockFactory=org.apache.lucene.store.NativeFSLockFactory@53925392),segFN=segments_1,generation=1}
[junit4:junit4] 2> 21506 T97 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 21515 T97 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 21517 T97 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 21519 T97 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 21521 T97 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 21524 T97 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 21525 T97 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 21528 T97 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 21531 T97 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 21533 T97 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 21536 T97 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 21541 T97 oass.SolrIndexSearcher.<init> Opening Searcher@2c592c59 main
[junit4:junit4] 2> 21542 T97 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty3/tlog
[junit4:junit4] 2> 21545 T97 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 21546 T97 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 21582 T98 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2c592c59 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 21587 T97 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 21588 T97 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51603 collection:collection1 shard:shard1
[junit4:junit4] 2> 21603 T97 oasc.ZkController.register We are http://127.0.0.1:51603/collection1/ and leader is http://127.0.0.1:50340/collection1/
[junit4:junit4] 2> 21604 T97 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51603
[junit4:junit4] 2> 21605 T97 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 21606 T97 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@31923192 url=http://127.0.0.1:51603/collection1 node=127.0.0.1:51603_ C0_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:51603, state=down, node_name=127.0.0.1:51603_, collection=collection1, core=collection1}
[junit4:junit4] 2> 21611 T99 C0 P51603 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 21611 T97 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 21613 T99 C0 P51603 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 21616 T99 C0 P51603 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 21617 T99 C0 P51603 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 21618 T18 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 21619 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 21622 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 21633 T99 C0 P51603 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 21885 T54 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:51603__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 21985 T18 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 21989 T18 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55265
[junit4:junit4] 2> 21991 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 21992 T18 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 21994 T18 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372548620458
[junit4:junit4] 2> 21995 T18 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372548620458/solr.xml
[junit4:junit4] 2> 21996 T18 oasc.CoreContainer.<init> New CoreContainer 1546214441
[junit4:junit4] 2> 21997 T18 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372548620458/'
[junit4:junit4] 2> 21998 T18 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372548620458/'
[junit4:junit4] 2> 22082 T18 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 22083 T18 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 22084 T18 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 22085 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 22086 T18 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 22086 T18 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 22087 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 22088 T18 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 22089 T18 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 22090 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 22097 T18 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 22099 T18 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42922/solr
[junit4:junit4] 2> 22100 T18 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 22102 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 22105 T111 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15e815e8 name:ZooKeeperConnection Watcher:127.0.0.1:42922 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 22106 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 22110 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 22118 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 22121 T113 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@509c509c name:ZooKeeperConnection Watcher:127.0.0.1:42922/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 22122 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 22131 T18 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 22718 T45 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 22720 T45 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51603",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:51603__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:51603_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 22743 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 22743 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 22743 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 22743 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 22743 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 22743 T51 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 22891 T54 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:51603__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 22892 T54 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={state=recovering&onlyIfLeader=true&checkLive=true&action=PREPRECOVERY&version=2&coreNodeName=127.0.0.1:51603__collection1&wt=javabin&nodeName=127.0.0.1:51603_&core=collection1} status=0 QTime=1012
[junit4:junit4] 2> 23154 T18 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55265_
[junit4:junit4] 2> 23157 T18 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55265_
[junit4:junit4] 2> 23161 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 23163 T80 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 23163 T64 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 23165 T51 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 23165 T96 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 23166 T51 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 23167 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 23168 T113 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 23165 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 23164 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 23169 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 23174 T44 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 23185 T114 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 23186 T114 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 24266 T45 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 24268 T45 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55265",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55265_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 24269 T45 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 24269 T45 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 24280 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 24280 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 24281 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 24281 T51 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 24281 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 24285 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> ASYNC NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@31923192 url=http://127.0.0.1:51603/collection1 node=127.0.0.1:51603_ C1_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:51603, state=recovering, node_name=127.0.0.1:51603_, collection=collection1, shard=shard1, core=collection1}
[junit4:junit4] 2> 24916 T99 C1 P51603 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:50340/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 24920 T99 C1 P51603 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 24934 T99 C1 P51603 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:51603 START replicas=[http://127.0.0.1:50340/collection1/] nUpdates=100
[junit4:junit4] 2> 24936 T99 C1 P51603 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 24937 T99 C1 P51603 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 24938 T99 C1 P51603 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 24938 T99 C1 P51603 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 24939 T99 C1 P51603 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 24940 T99 C1 P51603 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:50340/collection1/. core=collection1
[junit4:junit4] 2> 24941 T99 C1 P51603 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@14801480 url=http://127.0.0.1:50340/collection1 node=127.0.0.1:50340_ C2_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:50340, state=active, node_name=127.0.0.1:50340_, collection=collection1, shard=shard1, core=collection1, leader=true}
[junit4:junit4] 2> 24956 T55 C2 P50340 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&qt=/get&distrib=false&version=2&wt=javabin} status=0 QTime=3
[junit4:junit4] 2> 24972 T60 C2 P50340 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 24977 T60 C2 P50340 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2cba2cba lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d765d76),segFN=segments_1,generation=1}
[junit4:junit4] 2> 24978 T60 C2 P50340 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 24980 T60 C2 P50340 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 24982 T60 C2 P50340 oass.SolrIndexSearcher.<init> Opening Searcher@10541054 realtime
[junit4:junit4] 2> 24983 T60 C2 P50340 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 24984 T60 C2 P50340 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={softCommit=false&commit_end_point=true&commit=true&version=2&waitSearcher=true&wt=javabin&openSearcher=false} {commit=} 0 26
[junit4:junit4] 2> 24999 T99 C1 P51603 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 25001 T99 C1 P51603 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4] 2> 25013 T99 C1 P51603 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> ASYNC NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@14801480 url=http://127.0.0.1:50340/collection1 node=127.0.0.1:50340_ C3_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:50340, state=active, node_name=127.0.0.1:50340_, collection=collection1, shard=shard1, core=collection1, leader=true}
[junit4:junit4] 2> 25085 T55 C3 P50340 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 25087 T55 C3 P50340 oasc.SolrCore.execute [collection1] webapp= path=/replication params={qt=/replication&version=2&command=indexversion&wt=javabin} status=0 QTime=68
[junit4:junit4] 2> ASYNC NEW_CORE C4 name=collection1 org.apache.solr.core.SolrCore@31923192 url=http://127.0.0.1:51603/collection1 node=127.0.0.1:51603_ C4_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:51603, state=recovering, node_name=127.0.0.1:51603_, collection=collection1, shard=shard1, core=collection1}
[junit4:junit4] 2> 25090 T99 C4 P51603 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 25090 T99 C4 P51603 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 25091 T99 C4 P51603 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 25091 T99 C4 P51603 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 25094 T99 C4 P51603 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 25189 T114 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372548620458/collection1
[junit4:junit4] 2> 25190 T114 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 25193 T114 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 25194 T114 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 25196 T114 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372548620458/collection1/'
[junit4:junit4] 2> 25198 T114 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372548620458/collection1/lib/README' to classloader
[junit4:junit4] 2> 25199 T114 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372548620458/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 25239 T114 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 25288 T114 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 25291 T114 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 25300 T114 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 25800 T45 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 25802 T45 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51603",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:51603__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:51603_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 25814 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 25814 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 25814 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 25814 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 25814 T51 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 25814 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 25859 T114 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 25860 T114 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 25861 T114 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 25873 T114 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 25879 T114 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 25927 T114 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 25937 T114 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 25947 T114 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 25950 T114 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 25951 T114 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 25952 T114 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 25955 T114 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 25969 T114 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 25971 T114 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 25971 T114 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1372548620458/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty4/
[junit4:junit4] 2> 25973 T114 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56ac56ac
[junit4:junit4] 2> 25974 T114 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 25976 T114 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty4
[junit4:junit4] 2> 25977 T114 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty4/index/
[junit4:junit4] 2> 25977 T114 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 25979 T114 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty4/index
[junit4:junit4] 2> 25988 T114 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7bbe7bbe lockFactory=org.apache.lucene.store.NativeFSLockFactory@4eb04eb),segFN=segments_1,generation=1}
[junit4:junit4] 2> 25989 T114 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 25995 T114 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 25996 T114 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 25998 T114 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 26000 T114 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 26002 T114 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 26002 T114 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 26006 T114 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 26008 T114 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 26009 T114 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 26012 T114 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 26016 T114 oass.SolrIndexSearcher.<init> Opening Searcher@3aa73aa7 main
[junit4:junit4] 2> 26017 T114 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty4/tlog
[junit4:junit4] 2> 26019 T114 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 26021 T114 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 26034 T116 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3aa73aa7 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 26043 T114 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 26044 T114 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55265 collection:collection1 shard:shard2
[junit4:junit4] 2> 26053 T114 oasc.ZkController.register We are http://127.0.0.1:55265/collection1/ and leader is http://127.0.0.1:40848/collection1/
[junit4:junit4] 2> 26054 T114 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55265
[junit4:junit4] 2> 26055 T114 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 26056 T114 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> 26058 T114 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C5 name=collection1 org.apache.solr.core.SolrCore@60916091 url=http://127.0.0.1:55265/collection1 node=127.0.0.1:55265_ C5_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:55265, state=down, node_name=127.0.0.1:55265_, collection=collection1, core=collection1}
[junit4:junit4] 2> 26059 T117 C5 P55265 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 26060 T117 C5 P55265 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 26061 T117 C5 P55265 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 26062 T18 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 26063 T18 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 26065 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 26062 T117 C5 P55265 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 26072 T117 C5 P55265 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 26077 T18 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 26082 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 26111 T74 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:55265__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 26125 T60 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4] 2> 27345 T45 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 27348 T45 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55265",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:55265__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:55265_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 27360 T45 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50340",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50340_",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 27361 T45 oasc.Overseer$ClusterStateUpdater.createCollection Create collection onenodecollection with numShards 1
[junit4:junit4] 2> 27362 T45 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 27370 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 27371 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 27371 T51 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 27371 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 27371 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 27375 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 28112 T74 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:55265__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 28113 T74 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={state=recovering&onlyIfLeader=true&checkLive=true&action=PREPRECOVERY&version=2&coreNodeName=127.0.0.1:55265__collection1&wt=javabin&nodeName=127.0.0.1:55265_&core=collection1} status=0 QTime=2003
[junit4:junit4] 2> 28128 T60 oasc.CoreContainer.create Creating SolrCore 'onenodecollectioncore' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372548606209/onenodecollectioncore
[junit4:junit4] 2> 28130 T60 oasc.ZkController.createCollectionZkNode Check for collection zkNode:onenodecollection
[junit4:junit4] 2> 28132 T60 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:onenodecollection
[junit4:junit4] 2> 28133 T60 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4] 2> 28136 T60 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4] 2> 28138 T60 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection
[junit4:junit4] 2> 28144 T60 oasc.ZkController.readConfigName Load collection config from:/collections/onenodecollection
[junit4:junit4] 2> 28148 T60 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372548606209/onenodecollectioncore/'
[junit4:junit4] 2> 28255 T60 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 28332 T60 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 28335 T60 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 28342 T60 oass.IndexSchema.readSchema [onenodecollectioncore] Schema name=test
[junit4:junit4] 2> 28934 T60 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 28935 T60 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 28937 T60 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 28949 T60 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 28955 T60 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 29011 T60 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 29022 T60 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 29036 T60 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 29040 T60 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 29041 T60 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 29042 T60 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 29045 T60 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 29046 T60 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 29047 T60 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 29048 T60 oasc.SolrCore.<init> [onenodecollectioncore] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1372548606209/onenodecollectioncore/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372548599286/onenodecollection/
[junit4:junit4] 2> 29049 T60 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56ac56ac
[junit4:junit4] 2> 29051 T60 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 29052 T60 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372548599286/onenodecollection
[junit4:junit4] 2> 29053 T60 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372548599286/onenodecollection/index/
[junit4:junit4] 2> 29055 T60 oasc.SolrCore.initIndex WARN [onenodecollectioncore] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372548599286/onenodecollection/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 29056 T60 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372548599286/onenodecollection/index
[junit4:junit4] 2> 29062 T60 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@15d415d4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@159b159b),segFN=segments_1,generation=1}
[junit4:junit4] 2> 29063 T60 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 29070 T60 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 29071 T60 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 29072 T60 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 29073 T60 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 29075 T60 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 29075 T60 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 29077 T60 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 29078 T60 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 29079 T60 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 29081 T60 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 29084 T60 oass.SolrIndexSearcher.<init> Opening Searcher@1c4d1c4d main
[junit4:junit4] 2> 29085 T60 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372548599286/onenodecollection/tlog
[junit4:junit4] 2> 29087 T60 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 29088 T60 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 29102 T119 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@1c4d1c4d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 29118 T60 oasc.CoreContainer.registerCore registering core: onenodecollectioncore
[junit4:junit4] 2> 29119 T60 oasc.ZkController.register Register replica - core:onenodecollectioncore address:http://127.0.0.1:50340 collection:onenodecollection shard:shard1
[junit4:junit4] 2> 29121 T60 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leader_elect/shard1/election
[junit4:junit4] 2> 29136 T60 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 29146 T60 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 29147 T60 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 29147 T60 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50340/onenodecollectioncore/
[junit4:junit4] 2> 29148 T60 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 29149 T60 oasc.SyncStrategy.syncToMe http://127.0.0.1:50340/onenodecollectioncore/ has no replicas
[junit4:junit4] 2> 29150 T60 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50340/onenodecollectioncore/
[junit4:junit4] 2> 29150 T60 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leaders/shard1
[junit4:junit4] 2> ASYNC NEW_CORE C6 name=collection1 org.apache.solr.core.SolrCore@60916091 url=http://127.0.0.1:55265/collection1 node=127.0.0.1:55265_ C6_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:55265, state=recovering, node_name=127.0.0.1:55265_, collection=collection1, shard=shard2, core=collection1}
[junit4:junit4] 2> 30115 T117 C6 P55265 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:40848/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 30116 T117 C6 P55265 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55265 START replicas=[http://127.0.0.1:40848/collection1/] nUpdates=100
[junit4:junit4] 2> 30117 T117 C6 P55265 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 30118 T117 C6 P55265 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 30118 T117 C6 P55265 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 30119 T117 C6 P55265 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 30120 T117 C6 P55265 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 30120 T117 C6 P55265 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:40848/collection1/. core=collection1
[junit4:junit4] 2> 30121 T117 C6 P55265 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C7 name=collection1 org.apache.solr.core.SolrCore@6f4e6f4e url=http://127.0.0.1:40848/collection1 node=127.0.0.1:40848_ C7_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:40848, state=active, node_name=127.0.0.1:40848_, collection=collection1, shard=shard2, core=collection1, leader=true}
[junit4:junit4] 2> 30128 T73 C7 P40848 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&qt=/get&distrib=false&version=2&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 30133 T76 C7 P40848 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 30137 T76 C7 P40848 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@21622162 lockFactory=org.apache.lucene.store.NativeFSLockFactory@17bd17bd),segFN=segments_1,generation=1}
[junit4:junit4] 2> 30137 T76 C7 P40848 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 30138 T76 C7 P40848 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 30139 T76 C7 P40848 oass.SolrIndexSearcher.<init> Opening Searcher@59a559a5 realtime
[junit4:junit4] 2> 30140 T76 C7 P40848 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 30141 T76 C7 P40848 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={softCommit=false&commit_end_point=true&commit=true&version=2&waitSearcher=true&wt=javabin&openSearcher=false} {commit=} 0 8
[junit4:junit4] 2> 30143 T117 C6 P55265 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 30143 T117 C6 P55265 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 30147 T73 C7 P40848 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 30149 T73 C7 P40848 oasc.SolrCore.execute [collection1] webapp= path=/replication params={qt=/replication&version=2&command=indexversion&wt=javabin} status=0 QTime=2
[junit4:junit4] 2> 30151 T117 C6 P55265 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 30152 T117 C6 P55265 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 30152 T117 C6 P55265 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 30153 T117 C6 P55265 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 30156 T117 C6 P55265 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 30398 T45 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 30484 T45 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55265",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:55265__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:55265_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 30491 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 30492 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 30491 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 30492 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 30492 T51 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 30491 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 30533 T60 oasc.ZkController.register We are http://127.0.0.1:50340/onenodecollectioncore/ and leader is http://127.0.0.1:50340/onenodecollectioncore/
[junit4:junit4] 2> 30534 T60 oasc.ZkController.register No LogReplay needed for core=onenodecollectioncore baseURL=http://127.0.0.1:50340
[junit4:junit4] 2> 30535 T60 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 30536 T60 oasc.ZkController.publish publishing core=onenodecollectioncore state=active
[junit4:junit4] 2> 30540 T60 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 30542 T60 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372548599286/onenodecollection&action=CREATE&version=2&numShards=1&name=onenodecollectioncore&wt=javabin&roles=none&collection=onenodecollection} status=0 QTime=4421
[junit4:junit4] 2> 30544 T18 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: onenodecollection failOnTimeout:true timeout (sec):230
[junit4:junit4] 2> 30545 T18 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 31548 T18 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 32014 T45 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 32017 T45 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50340",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:50340__onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:50340_",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 32028 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 32028 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 32028 T51 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 32028 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 32028 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 32028 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 32552 T18 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 32555 T18 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: onenodecollection
[junit4:junit4] 2> 32557 T18 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 32586 T18 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 32590 T122 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a620a62 name:ZooKeeperConnection Watcher:127.0.0.1:42922 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 32591 T18 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 32766 T18 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 50832
[junit4:junit4] 2> 32767 T18 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1569807761
[junit4:junit4] 2> 33553 T45 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 33555 T45 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50832",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:50832__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:50832_",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 33562 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 33562 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 33562 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 33563 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 33562 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34773 T18 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 34774 T18 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 34777 T18 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@54be54be
[junit4:junit4] 2> 34784 T18 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 34785 T18 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 34786 T18 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 34787 T18 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 34788 T18 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 34789 T18 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 34790 T18 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/control/data/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/control/data/index;done=false>>]
[junit4:junit4] 2> 34791 T18 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/control/data/index
[junit4:junit4] 2> 34792 T18 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/control/data [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/control/data;done=false>>]
[junit4:junit4] 2> 34792 T18 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/control/data
[junit4:junit4] 2> 34794 T45 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89951345013161987-127.0.0.1:50832_-n_0000000000) am no longer a leader.
[junit4:junit4] 2> 34797 T80 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 34797 T64 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 34798 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 34798 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 34799 T96 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 34801 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 34798 T113 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 34799 T44 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 34805 T44 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 34806 T44 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 34806 T64 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 34804 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 34814 T64 oasc.Overseer.start Overseer (id=89951345013161990-127.0.0.1:50340_-n_0000000001) starting
[junit4:junit4] 2> 34823 T18 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 34829 T123 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 34829 T124 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 34837 T123 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 34839 T123 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50832",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:50832__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:50832_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 34932 T18 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 50340
[junit4:junit4] 2> 34932 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 34932 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 34934 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 34933 T18 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1899917630
[junit4:junit4] 2> 34933 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 36457 T123 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 36458 T123 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50340",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:50340__onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:50340_",
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 36470 T123 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50340",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:50340__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:50340_",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 36481 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 36481 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 36481 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 36481 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 37947 T18 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 37948 T18 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 37951 T18 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4] 2> 37957 T18 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@14801480
[junit4:junit4] 2> 37977 T18 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 37978 T18 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 37979 T18 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 37979 T18 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 37981 T18 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 37982 T18 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 37982 T18 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty1 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty1;done=false>>]
[junit4:junit4] 2> 37983 T18 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty1
[junit4:junit4] 2> 37984 T18 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty1/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty1/index;done=false>>]
[junit4:junit4] 2> 37984 T18 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty1/index
[junit4:junit4] 2> 37985 T18 oasc.SolrCore.close [onenodecollectioncore] CLOSING SolrCore org.apache.solr.core.SolrCore@48814881
[junit4:junit4] 2> 37995 T18 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 37996 T18 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 37997 T18 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 37997 T18 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 37999 T123 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 37999 T18 oasc.SolrCore.closeSearcher [onenodecollectioncore] Closing main searcher on request.
[junit4:junit4] 2> 38000 T18 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 38000 T123 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50340",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:50340__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:50340_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 38001 T18 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372548599286/onenodecollection/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372548599286/onenodecollection/index;done=false>>]
[junit4:junit4] 2> 38002 T18 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372548599286/onenodecollection/index
[junit4:junit4] 2> 38003 T18 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372548599286/onenodecollection [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372548599286/onenodecollection;done=false>>]
[junit4:junit4] 2> 38004 T18 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1372548599286/onenodecollection
[junit4:junit4] 2> 38009 T96 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 38009 T113 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 38010 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 38011 T80 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 38010 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 38018 T96 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 38018 T80 oasc.Overseer.start Overseer (id=89951345013161992-127.0.0.1:40848_-n_0000000002) starting
[junit4:junit4] 2> 38027 T96 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 38027 T126 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 38028 T96 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 38030 T96 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 38043 T18 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 38043 T96 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51603/collection1/
[junit4:junit4] 2> 38029 T80 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 38045 T96 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 38047 T96 oasc.SyncStrategy.syncToMe http://127.0.0.1:51603/collection1/ has no replicas
[junit4:junit4] 2> 38046 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 38048 T96 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51603/collection1/
[junit4:junit4] 2> 38050 T96 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 38051 T125 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 38053 T125 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
[junit4:junit4] 2> 38056 T125 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50340",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:50340__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:50340_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 38058 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 38066 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 38066 T125 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 38071 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 38074 T125 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 38076 T125 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50340",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:50340__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:50340_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 38085 T125 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50340",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:50340__onenodecollectioncore",
[junit4:junit4] 2> "node_name":"127.0.0.1:50340_",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":"none",
[junit4:junit4] 2> "collection":"onenodecollection",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"onenodecollectioncore"}
[junit4:junit4] 2> 38096 T18 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 40848
[junit4:junit4] 2> 38097 T18 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=646129283
[junit4:junit4] 2> 38126 T125 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40848",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:40848__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:40848_",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 38152 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 38152 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 38175 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 39101 T18 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 39102 T18 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 39105 T18 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@6f4e6f4e
[junit4:junit4] 2> 39114 T18 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 39115 T18 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 39116 T18 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 39117 T18 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 39121 T18 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 39122 T18 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 39123 T18 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty2/index;done=false>>]
[junit4:junit4] 2> 39124 T18 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty2/index
[junit4:junit4] 2> 39125 T18 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty2;done=false>>]
[junit4:junit4] 2> 39129 T18 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty2
[junit4:junit4] 2> 39133 T125 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89951345013161992-127.0.0.1:40848_-n_0000000002) am no longer a leader.
[junit4:junit4] 2> 39138 T80 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 39139 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 39140 T80 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 39140 T113 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 39142 T96 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 39147 T113 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4] 2> 39148 T113 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4] 2> 39149 T113 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 39149 T96 oasc.Overseer.start Overseer (id=89951345013161994-127.0.0.1:51603_-n_0000000003) starting
[junit4:junit4] 2> 39150 T113 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55265/collection1/
[junit4:junit4] 2> 39152 T113 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55265 START replicas=[http://127.0.0.1:40848/collection1/] nUpdates=100
[junit4:junit4] 2> 39153 T113 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55265 DONE. We have no versions. sync failed.
[junit4:junit4] 2> 39154 T113 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
[junit4:junit4] 2> 39155 T113 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4:junit4] 2> 39155 T113 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55265/collection1/
[junit4:junit4] 2> 39156 T113 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 39222 T18 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 39233 T129 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 39236 T96 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 39237 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 39239 T113 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 39240 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 39242 T128 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 39252 T128 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 39254 T128 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40848",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:40848__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:40848_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 39274 T18 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 51603
[junit4:junit4] 2> 39275 T18 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=900412843
[junit4:junit4] 2> 39285 T128 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51603",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:51603__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:51603_",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 39294 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 39297 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 39316 T64 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 39317 T64 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 39318 T64 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 40280 T18 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 40281 T18 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 40284 T18 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@31923192
[junit4:junit4] 2> 40290 T18 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 40292 T18 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 40292 T18 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 40293 T18 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 40294 T18 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 40294 T18 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 40295 T18 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty3/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty3/index;done=false>>]
[junit4:junit4] 2> 40296 T18 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty3/index
[junit4:junit4] 2> 40296 T18 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty3 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty3;done=false>>]
[junit4:junit4] 2> 40297 T18 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty3
[junit4:junit4] 2> 40298 T128 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89951345013161994-127.0.0.1:51603_-n_0000000003) am no longer a leader.
[junit4:junit4] 2> 40301 T113 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4] 2> 40301 T96 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 40302 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 40303 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 40304 T96 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 40308 T113 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 40312 T113 oasc.Overseer.start Overseer (id=89951345013161996-127.0.0.1:55265_-n_0000000004) starting
[junit4:junit4] 2> 40321 T131 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 40322 T18 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 40326 T130 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 40331 T130 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 40332 T130 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51603",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:51603__collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:51603_",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "core":"collection1"}
[junit4:junit4] 2> 40348 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 40376 T18 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 55265
[junit4:junit4] 2> 40376 T18 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1546214441
[junit4:junit4] 2> 41379 T18 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 41379 T18 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 41382 T18 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@60916091
[junit4:junit4] 2> 41393 T18 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 41394 T18 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 41394 T18 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 41395 T18 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 41396 T18 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 41397 T18 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 41397 T18 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty4 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty4;done=false>>]
[junit4:junit4] 2> 41398 T18 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty4
[junit4:junit4] 2> 41398 T18 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty4/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty4/index;done=false>>]
[junit4:junit4] 2> 41399 T18 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.BasicDistributedZk2Test-1372548599284/jetty4/index
[junit4:junit4] 2> 41400 T130 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89951345013161996-127.0.0.1:55265_-n_0000000004) am no longer a leader.
[junit4:junit4] 2> 41423 T18 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 41484 T18 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 41487 T18 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:42922 42922
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> / (2)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr (7)
[junit4:junit4] 1> /solr/configs (1)
[junit4:junit4] 1> /solr/configs/conf1 (9)
[junit4:junit4] 1> /solr/configs/conf1/old_synonyms.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/protwords.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/stopwords.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/synonyms.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/schema.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/currency.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/open-exchange-rates.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "disclaimer": "This data is not real, it was synthetically created to match currency.xml. It is modeled after the data format available from openexchangerates.org. See https://openexchangerates.org/documentation for details",
[junit4:junit4] 1> "license": "http://www.apache.org/licenses/LICENSE-2.0",
[junit4:junit4] 1> "timestamp": 1332070464,
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 1> "IMPORTANT NOTE": "In order for tests to work, this data must be kept in sync with ./currency.xml",
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 1> "base": "USD",
[junit4:junit4] 1> "rates": {
[junit4:junit4] 1> "USD": 1,
[junit4:junit4] 1> "JPY": 81.29,
[junit4:junit4] 1> "EUR": 2.5,
[junit4:junit4] 1> "GBP": 0.5,
[junit4:junit4] 1> "MXN": 2.0
[junit4:junit4] 1> }
[junit4:junit4] 1> }
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/configs/conf1/solrconfig.xml (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/configs/conf1/mapping-ISOLatin1Accent.txt (0)
[junit4:junit4] 1> DATA: ...supressed...
[junit4:junit4] 1> /solr/overseer (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1> /solr/overseer/queue (0)
[junit4:junit4] 1> /solr/overseer/collection-queue-work (0)
[junit4:junit4] 1> /solr/overseer/queue-work (0)
[junit4:junit4] 1> /solr/overseer_elect (2)
[junit4:junit4] 1> /solr/overseer_elect/leader (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"id":"89951345013161987-127.0.0.1:50832_-n_0000000000"}
[junit4:junit4] 1> /solr/overseer_elect/election (5)
[junit4:junit4] 1> /solr/overseer_elect/election/89951345013161990-127.0.0.1:50340_-n_0000000001 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89951345013161996-127.0.0.1:55265_-n_0000000004 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89951345013161994-127.0.0.1:51603_-n_0000000003 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89951345013161992-127.0.0.1:40848_-n_0000000002 (0)
[junit4:junit4] 1> /solr/overseer_elect/election/89951345013161987-127.0.0.1:50832_-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections (3)
[junit4:junit4] 1> /solr/collections/control_collection (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/89951345013161987-127.0.0.1:50832__collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:50832",
[junit4:junit4] 1> "node_name":"127.0.0.1:50832_",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/control_collection/shards (0)
[junit4:junit4] 1> /solr/collections/collection1 (3)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {"configName":"conf1"}
[junit4:junit4] 1> /solr/collections/collection1/leader_elect (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89951345013161994-127.0.0.1:51603__collection1-n_0000000001 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/89951345013161990-127.0.0.1:50340__collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2 (1)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election (2)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election/89951345013161992-127.0.0.1:40848__collection1-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/collection1/leader_elect/shard2/election/89951345013161996-127.0.0.1:55265__collection1-n_0000000001 (0)
[junit4:junit4] 1> /solr/collections/collection1/leaders (2)
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:50340",
[junit4:junit4] 1> "node_name":"127.0.0.1:50340_",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/leaders/shard2 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:40848",
[junit4:junit4] 1> "node_name":"127.0.0.1:40848_",
[junit4:junit4] 1> "core":"collection1"}
[junit4:junit4] 1> /solr/collections/collection1/shards (0)
[junit4:junit4] 1> /solr/collections/onenodecollection (2)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "router":"compositeId",
[junit4:junit4] 1> "configName":"conf1"}
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1 (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1/election (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leader_elect/shard1/election/89951345013161990-127.0.0.1:50340__onenodecollectioncore-n_0000000000 (0)
[junit4:junit4] 1> /solr/collections/onenodecollection/leaders (1)
[junit4:junit4] 1> /solr/collections/onenodecollection/leaders/shard1 (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "base_url":"http://127.0.0.1:50340",
[junit4:junit4] 1> "node_name":"127.0.0.1:50340_",
[junit4:junit4] 1> "core":"onenodecollectioncore"}
[junit4:junit4] 1> /solr/aliases.json (0)
[junit4:junit4] 1> /solr/live_nodes (5)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:50340_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:50832_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:55265_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:40848_ (0)
[junit4:junit4] 1> /solr/live_nodes/127.0.0.1:51603_ (0)
[junit4:junit4] 1> /solr/clusterstate.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "control_collection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"127.0.0.1:50832__collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:50832",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:50832_",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "onenodecollection":{
[junit4:junit4] 1> "shards":{"shard1":{
[junit4:junit4] 1> "range":"80000000-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{"127.0.0.1:50340__onenodecollectioncore":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:50340",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:50340_",
[junit4:junit4] 1> "roles":"none",
[junit4:junit4] 1> "collection":"onenodecollection",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"onenodecollectioncore",
[junit4:junit4] 1> "leader":"true"}}}},
[junit4:junit4] 1> "router":"compositeId"},
[junit4:junit4] 1> "collection1":{
[junit4:junit4] 1> "shards":{
[junit4:junit4] 1> "shard1":{
[junit4:junit4] 1> "range":"80000000-ffffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{
[junit4:junit4] 1> "127.0.0.1:50340__collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:50340",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:50340_",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"},
[junit4:junit4] 1> "127.0.0.1:51603__collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:51603",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:51603_",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "core":"collection1"}}},
[junit4:junit4] 1> "shard2":{
[junit4:junit4] 1> "range":"0-7fffffff",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "replicas":{
[junit4:junit4] 1> "127.0.0.1:40848__collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:40848",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:40848_",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "leader":"true"},
[junit4:junit4] 1> "127.0.0.1:55265__collection1":{
[junit4:junit4] 1> "base_url":"http://127.0.0.1:55265",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "node_name":"127.0.0.1:55265_",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "core":"collection1"}}}},
[junit4:junit4] 1> "router":"compositeId"}}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=BasicDistributedZk2Test -Dtests.method=testDistribSearch -Dtests.seed=CA85B2609398554B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=fr_BE_PREEURO -Dtests.timezone=Europe/Stockholm -Dtests.file.encoding=US-ASCII
[junit4:junit4] ERROR 41.1s J1 | BasicDistributedZk2Test.testDistribSearch <<<
[junit4:junit4] > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:40848/onenodecollectioncore returned non ok status:404, message:Can not find: /onenodecollectioncore/update
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([CA85B2609398554B:4B633C78E4C73577]:0)
[junit4:junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
[junit4:junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4] > at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
[junit4:junit4] > at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
[junit4:junit4] > at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
[junit4:junit4] > at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:198)
[junit4:junit4] > at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:90)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:738)
[junit4:junit4] 2> 41541 T18 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 41215 T17 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 42702 T113 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 42703 T113 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 42704 T113 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {}, docValues:{}, sim=DefaultSimilarity, locale=fr_BE_PREEURO, timezone=Europe/Stockholm
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic x86/IBM Corporation 1.6.0 (32-bit)/cpus=8,threads=3,free=15993528,total=30927360
[junit4:junit4] 2> NOTE: All tests run in this JVM: [BasicDistributedZk2Test]
[junit4:junit4] Completed on J1 in 43.03s, 1 test, 1 error <<< FAILURES!
[...truncated 914 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:385: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1248: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:892: There were test failures: 299 suites, 1290 tests, 1 error, 13 ignored (7 assumptions)
Total time: 48 minutes 42 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/ibm-j9-jdk6 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0-ea-b94) - Build #
6298 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6298/
Java: 32bit/jdk1.8.0-ea-b94 -client -XX:+UseParallelGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
Error Message:
9 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest: 1) Thread[id=4085, name=qtp6705546-4085, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2077) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:724) 2) Thread[id=4086, name=qtp6705546-4086, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2077) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:724) 3) Thread[id=4081, name=qtp6705546-4081 Acceptor0 SelectChannelConnector@127.0.0.1:53032, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:724) 4) Thread[id=4080, name=qtp6705546-4080 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:257) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:724) 5) Thread[id=4078, name=HashSessionScavenger-123, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) 6) Thread[id=4082, name=qtp6705546-4082 Acceptor1 SelectChannelConnector@127.0.0.1:53032, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210) at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:724) 7) Thread[id=4083, name=qtp6705546-4083, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2077) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:724) 8) Thread[id=4084, name=qtp6705546-4084, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2077) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang.Thread.run(Thread.java:724) 9) Thread[id=4079, name=qtp6705546-4079 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:257) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:724)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 9 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest:
1) Thread[id=4085, name=qtp6705546-4085, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2077)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:724)
2) Thread[id=4086, name=qtp6705546-4086, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2077)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:724)
3) Thread[id=4081, name=qtp6705546-4081 Acceptor0 SelectChannelConnector@127.0.0.1:53032, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)
4) Thread[id=4080, name=qtp6705546-4080 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:257)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)
5) Thread[id=4078, name=HashSessionScavenger-123, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
6) Thread[id=4082, name=qtp6705546-4082 Acceptor1 SelectChannelConnector@127.0.0.1:53032, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)
7) Thread[id=4083, name=qtp6705546-4083, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2077)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:724)
8) Thread[id=4084, name=qtp6705546-4084, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2077)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:724)
9) Thread[id=4079, name=qtp6705546-4079 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:257)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)
at __randomizedtesting.SeedInfo.seed([4F95D3036EBA0AEA]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=4081, name=qtp6705546-4081 Acceptor0 SelectChannelConnector@127.0.0.1:53032, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest] at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:724) 2) Thread[id=4080, name=qtp6705546-4080 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:257) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:724) 3) Thread[id=4078, name=HashSessionScavenger-123, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest] at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) 4) Thread[id=4082, name=qtp6705546-4082 Acceptor1 SelectChannelConnector@127.0.0.1:53032, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest] at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:724) 5) Thread[id=4079, name=qtp6705546-4079 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:257) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:724)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=4081, name=qtp6705546-4081 Acceptor0 SelectChannelConnector@127.0.0.1:53032, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)
2) Thread[id=4080, name=qtp6705546-4080 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:257)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)
3) Thread[id=4078, name=HashSessionScavenger-123, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
at java.util.TimerThread.run(Timer.java:505)
4) Thread[id=4082, name=qtp6705546-4082 Acceptor1 SelectChannelConnector@127.0.0.1:53032, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)
5) Thread[id=4079, name=qtp6705546-4079 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:257)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)
at __randomizedtesting.SeedInfo.seed([4F95D3036EBA0AEA]:0)
Build Log:
[...truncated 10111 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
[junit4:junit4] 2> 879861 T3877 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /pgum/m
[junit4:junit4] 2> 879865 T3877 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeyNothingIsSafeTest-1372554431074
[junit4:junit4] 2> 879866 T3877 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 879867 T3878 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 879967 T3877 oasc.ZkTestServer.run start zk server on port:42882
[junit4:junit4] 2> 879968 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 880034 T3884 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@86ddef name:ZooKeeperConnection Watcher:127.0.0.1:42882 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 880035 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 880035 T3877 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 880038 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 880039 T3886 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41380f name:ZooKeeperConnection Watcher:127.0.0.1:42882/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 880040 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 880040 T3877 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 880042 T3877 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 880044 T3877 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 880046 T3877 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 880047 T3877 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 880048 T3877 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 880051 T3877 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 880051 T3877 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 880054 T3877 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 880054 T3877 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 880056 T3877 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 880057 T3877 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 880059 T3877 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 880059 T3877 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 880061 T3877 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 880062 T3877 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 880064 T3877 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 880064 T3877 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 880066 T3877 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 880066 T3877 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 880068 T3877 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 880068 T3877 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 880145 T3877 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 880153 T3877 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48011
[junit4:junit4] 2> 880154 T3877 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 880155 T3877 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 880155 T3877 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372554431279
[junit4:junit4] 2> 880156 T3877 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372554431279/solr.xml
[junit4:junit4] 2> 880156 T3877 oasc.CoreContainer.<init> New CoreContainer 20237574
[junit4:junit4] 2> 880157 T3877 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372554431279/'
[junit4:junit4] 2> 880157 T3877 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372554431279/'
[junit4:junit4] 2> 880235 T3877 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 880236 T3877 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 880236 T3877 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 880237 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 880237 T3877 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 880238 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 880238 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 880239 T3877 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 880239 T3877 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 880240 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 880244 T3877 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 880245 T3877 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42882/solr
[junit4:junit4] 2> 880245 T3877 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 880246 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 880249 T3897 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12700a4 name:ZooKeeperConnection Watcher:127.0.0.1:42882 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 880250 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 880251 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 880256 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 880257 T3899 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5fc105 name:ZooKeeperConnection Watcher:127.0.0.1:42882/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 880257 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 880258 T3877 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 880261 T3877 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 880263 T3877 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 880265 T3877 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48011_pgum%2Fm
[junit4:junit4] 2> 880266 T3877 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48011_pgum%2Fm
[junit4:junit4] 2> 880268 T3877 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 880276 T3877 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 880279 T3877 oasc.Overseer.start Overseer (id=89951727195062275-127.0.0.1:48011_pgum%2Fm-n_0000000000) starting
[junit4:junit4] 2> 880281 T3877 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 880287 T3901 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 880287 T3877 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 880289 T3877 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 880291 T3877 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 880297 T3902 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 880297 T3900 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 880298 T3902 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 881801 T3900 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 881802 T3900 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:48011_pgum%2Fm",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48011/pgum/m"}
[junit4:junit4] 2> 881802 T3900 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 881802 T3900 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 881806 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 882299 T3902 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372554431279/collection1
[junit4:junit4] 2> 882300 T3902 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 882300 T3902 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 882300 T3902 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 882302 T3902 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372554431279/collection1/'
[junit4:junit4] 2> 882302 T3902 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372554431279/collection1/lib/README' to classloader
[junit4:junit4] 2> 882303 T3902 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372554431279/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 882339 T3902 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 882378 T3902 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 882379 T3902 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 882383 T3902 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 882761 T3902 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 882762 T3902 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 882762 T3902 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 882768 T3902 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 882771 T3902 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 882781 T3902 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 882784 T3902 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 882786 T3902 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 882787 T3902 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 882787 T3902 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 882788 T3902 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 882788 T3902 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 882789 T3902 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 882789 T3902 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 882789 T3902 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1372554431279/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/control/data/
[junit4:junit4] 2> 882789 T3902 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@9261c5
[junit4:junit4] 2> 882790 T3902 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 882790 T3902 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/control/data
[junit4:junit4] 2> 882790 T3902 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/control/data/index/
[junit4:junit4] 2> 882791 T3902 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 882791 T3902 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/control/data/index
[junit4:junit4] 2> 882804 T3902 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 882805 T3902 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 882806 T3902 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 882807 T3902 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 882807 T3902 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 882807 T3902 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 882808 T3902 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 882808 T3902 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 882808 T3902 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 882809 T3902 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 882809 T3902 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 882811 T3902 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 882814 T3902 oass.SolrIndexSearcher.<init> Opening Searcher@7ec9ae main
[junit4:junit4] 2> 882814 T3902 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 882814 T3902 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 882819 T3903 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7ec9ae main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 882820 T3902 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 882821 T3902 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48011/pgum/m collection:control_collection shard:shard1
[junit4:junit4] 2> 882822 T3902 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 882827 T3902 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 882830 T3902 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 882830 T3902 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 882830 T3902 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48011/pgum/m/collection1/
[junit4:junit4] 2> 882831 T3902 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 882831 T3902 oasc.SyncStrategy.syncToMe http://127.0.0.1:48011/pgum/m/collection1/ has no replicas
[junit4:junit4] 2> 882831 T3902 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48011/pgum/m/collection1/
[junit4:junit4] 2> 882831 T3902 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 883309 T3900 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 883316 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 883337 T3902 oasc.ZkController.register We are http://127.0.0.1:48011/pgum/m/collection1/ and leader is http://127.0.0.1:48011/pgum/m/collection1/
[junit4:junit4] 2> 883338 T3902 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48011/pgum/m
[junit4:junit4] 2> 883338 T3902 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 883338 T3902 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 883338 T3902 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 883341 T3902 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 883342 T3877 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 883342 T3877 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 883343 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 883346 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 883347 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 883348 T3906 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dc464f name:ZooKeeperConnection Watcher:127.0.0.1:42882/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 883348 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 883349 T3877 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 883351 T3877 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 883447 T3877 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 883449 T3877 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55954
[junit4:junit4] 2> 883449 T3877 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 883449 T3877 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 883450 T3877 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372554434560
[junit4:junit4] 2> 883450 T3877 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372554434560/solr.xml
[junit4:junit4] 2> 883450 T3877 oasc.CoreContainer.<init> New CoreContainer 17678609
[junit4:junit4] 2> 883451 T3877 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372554434560/'
[junit4:junit4] 2> 883451 T3877 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372554434560/'
[junit4:junit4] 2> 883508 T3877 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 883509 T3877 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 883509 T3877 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 883510 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 883510 T3877 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 883510 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 883511 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 883511 T3877 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 883511 T3877 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 883512 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 883515 T3877 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 883515 T3877 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42882/solr
[junit4:junit4] 2> 883516 T3877 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 883516 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 883517 T3917 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f7aca6 name:ZooKeeperConnection Watcher:127.0.0.1:42882 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 883518 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 883519 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 883522 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 883523 T3919 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1789d35 name:ZooKeeperConnection Watcher:127.0.0.1:42882/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 883523 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 883526 T3877 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 884528 T3877 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55954_pgum%2Fm
[junit4:junit4] 2> 884529 T3877 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55954_pgum%2Fm
[junit4:junit4] 2> 884531 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 884531 T3906 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 884532 T3919 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 884533 T3899 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 884536 T3920 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 884536 T3920 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 884821 T3900 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 884822 T3900 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:48011_pgum%2Fm",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48011/pgum/m"}
[junit4:junit4] 2> 884824 T3900 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55954_pgum%2Fm",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55954/pgum/m"}
[junit4:junit4] 2> 884824 T3900 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4] 2> 884824 T3900 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 884833 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 884833 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 884833 T3906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 885537 T3920 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372554434560/collection1
[junit4:junit4] 2> 885538 T3920 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 885539 T3920 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 885539 T3920 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 885540 T3920 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372554434560/collection1/'
[junit4:junit4] 2> 885540 T3920 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372554434560/collection1/lib/README' to classloader
[junit4:junit4] 2> 885541 T3920 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372554434560/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 885570 T3920 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 885606 T3920 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 885608 T3920 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 885612 T3920 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 885995 T3920 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 885996 T3920 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 885996 T3920 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 886002 T3920 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 886005 T3920 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 886015 T3920 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 886017 T3920 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 886020 T3920 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 886021 T3920 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 886021 T3920 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 886021 T3920 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 886022 T3920 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 886022 T3920 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 886022 T3920 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 886022 T3920 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1372554434560/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty1/
[junit4:junit4] 2> 886023 T3920 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@9261c5
[junit4:junit4] 2> 886023 T3920 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 886023 T3920 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty1
[junit4:junit4] 2> 886024 T3920 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty1/index/
[junit4:junit4] 2> 886024 T3920 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 886024 T3920 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty1/index
[junit4:junit4] 2> 886027 T3920 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 886027 T3920 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 886029 T3920 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 886029 T3920 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 886029 T3920 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 886030 T3920 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 886031 T3920 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 886031 T3920 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 886031 T3920 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 886032 T3920 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 886032 T3920 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 886034 T3920 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 886036 T3920 oass.SolrIndexSearcher.<init> Opening Searcher@16a40f main
[junit4:junit4] 2> 886037 T3920 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 886037 T3920 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 886042 T3921 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@16a40f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 886044 T3920 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 886044 T3920 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55954/pgum/m collection:collection1 shard:shard1
[junit4:junit4] 2> 886045 T3920 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 886050 T3920 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 886052 T3920 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 886052 T3920 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 886052 T3920 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55954/pgum/m/collection1/
[junit4:junit4] 2> 886052 T3920 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 886052 T3920 oasc.SyncStrategy.syncToMe http://127.0.0.1:55954/pgum/m/collection1/ has no replicas
[junit4:junit4] 2> 886053 T3920 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55954/pgum/m/collection1/
[junit4:junit4] 2> 886053 T3920 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 886339 T3900 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 886346 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 886346 T3906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 886346 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 886357 T3920 oasc.ZkController.register We are http://127.0.0.1:55954/pgum/m/collection1/ and leader is http://127.0.0.1:55954/pgum/m/collection1/
[junit4:junit4] 2> 886357 T3920 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55954/pgum/m
[junit4:junit4] 2> 886358 T3920 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 886358 T3920 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 886358 T3920 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 886359 T3920 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 886360 T3877 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 886361 T3877 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 886361 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 886445 T3877 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 886447 T3877 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53032
[junit4:junit4] 2> 886448 T3877 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 886448 T3877 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 886448 T3877 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372554437574
[junit4:junit4] 2> 886449 T3877 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372554437574/solr.xml
[junit4:junit4] 2> 886449 T3877 oasc.CoreContainer.<init> New CoreContainer 5235269
[junit4:junit4] 2> 886449 T3877 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372554437574/'
[junit4:junit4] 2> 886450 T3877 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372554437574/'
[junit4:junit4] 2> 886503 T3877 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 886504 T3877 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 886504 T3877 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 886505 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 886505 T3877 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 886505 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 886506 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 886506 T3877 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 886506 T3877 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 886507 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 886510 T3877 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 886510 T3877 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42882/solr
[junit4:junit4] 2> 886510 T3877 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 886511 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 886512 T3933 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18d50fe name:ZooKeeperConnection Watcher:127.0.0.1:42882 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 886513 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 886515 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 886518 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 886519 T3935 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@123665c name:ZooKeeperConnection Watcher:127.0.0.1:42882/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 886519 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 886521 T3877 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 887524 T3877 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53032_pgum%2Fm
[junit4:junit4] 2> 887526 T3877 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53032_pgum%2Fm
[junit4:junit4] 2> 887527 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 887528 T3919 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 887528 T3906 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 887528 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 887528 T3906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 887529 T3935 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 887529 T3899 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 887533 T3936 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 887533 T3936 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 887850 T3900 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 887851 T3900 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55954_pgum%2Fm",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55954/pgum/m"}
[junit4:junit4] 2> 887853 T3900 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53032_pgum%2Fm",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53032/pgum/m"}
[junit4:junit4] 2> 887854 T3900 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 887854 T3900 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 887856 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 887856 T3935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 887856 T3906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 887857 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 888534 T3936 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372554437574/collection1
[junit4:junit4] 2> 888535 T3936 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 888535 T3936 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 888536 T3936 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 888537 T3936 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372554437574/collection1/'
[junit4:junit4] 2> 888537 T3936 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372554437574/collection1/lib/README' to classloader
[junit4:junit4] 2> 888538 T3936 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372554437574/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 888573 T3936 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 888609 T3936 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 888610 T3936 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 888614 T3936 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 888993 T3936 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 888993 T3936 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 888994 T3936 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 889000 T3936 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 889002 T3936 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 889013 T3936 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 889016 T3936 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 889018 T3936 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 889019 T3936 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 889019 T3936 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 889019 T3936 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 889020 T3936 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 889020 T3936 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 889020 T3936 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 889021 T3936 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1372554437574/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty2/
[junit4:junit4] 2> 889021 T3936 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@9261c5
[junit4:junit4] 2> 889021 T3936 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 889022 T3936 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty2
[junit4:junit4] 2> 889022 T3936 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty2/index/
[junit4:junit4] 2> 889022 T3936 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 889022 T3936 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty2/index
[junit4:junit4] 2> 889026 T3936 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 889026 T3936 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 889027 T3936 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 889028 T3936 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 889028 T3936 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 889029 T3936 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 889030 T3936 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 889030 T3936 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 889030 T3936 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 889031 T3936 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 889031 T3936 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 889032 T3936 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 889035 T3936 oass.SolrIndexSearcher.<init> Opening Searcher@f981d9 main
[junit4:junit4] 2> 889036 T3936 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 889036 T3936 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 889042 T3937 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f981d9 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 889043 T3936 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 889044 T3936 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53032/pgum/m collection:collection1 shard:shard2
[junit4:junit4] 2> 889044 T3936 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 889049 T3936 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 889051 T3936 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 889051 T3936 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 889051 T3936 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53032/pgum/m/collection1/
[junit4:junit4] 2> 889051 T3936 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 889051 T3936 oasc.SyncStrategy.syncToMe http://127.0.0.1:53032/pgum/m/collection1/ has no replicas
[junit4:junit4] 2> 889052 T3936 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53032/pgum/m/collection1/
[junit4:junit4] 2> 889052 T3936 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 889362 T3900 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 889368 T3906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 889368 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 889368 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 889368 T3935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 889407 T3936 oasc.ZkController.register We are http://127.0.0.1:53032/pgum/m/collection1/ and leader is http://127.0.0.1:53032/pgum/m/collection1/
[junit4:junit4] 2> 889407 T3936 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53032/pgum/m
[junit4:junit4] 2> 889407 T3936 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 889408 T3936 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 889408 T3936 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 889409 T3936 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 889410 T3877 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 889411 T3877 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 889411 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 889480 T3877 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 889481 T3877 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52727
[junit4:junit4] 2> 889482 T3877 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 889482 T3877 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 889483 T3877 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372554440622
[junit4:junit4] 2> 889483 T3877 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372554440622/solr.xml
[junit4:junit4] 2> 889483 T3877 oasc.CoreContainer.<init> New CoreContainer 20921124
[junit4:junit4] 2> 889484 T3877 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372554440622/'
[junit4:junit4] 2> 889484 T3877 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372554440622/'
[junit4:junit4] 2> 889531 T3877 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 889532 T3877 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 889532 T3877 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 889533 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 889533 T3877 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 889533 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 889534 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 889534 T3877 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 889534 T3877 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 889535 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 889537 T3877 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 889538 T3877 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42882/solr
[junit4:junit4] 2> 889538 T3877 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 889539 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 889540 T3949 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69b7fb name:ZooKeeperConnection Watcher:127.0.0.1:42882 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 889541 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 889544 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 889546 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 889547 T3951 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ce4ff6 name:ZooKeeperConnection Watcher:127.0.0.1:42882/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 889548 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 889551 T3877 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 890553 T3877 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52727_pgum%2Fm
[junit4:junit4] 2> 890555 T3877 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52727_pgum%2Fm
[junit4:junit4] 2> 890556 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 890557 T3906 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 890557 T3919 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 890557 T3951 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 890557 T3935 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 890557 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 890557 T3906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 890558 T3899 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 890558 T3935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 890561 T3952 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 890562 T3952 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 890873 T3900 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 890873 T3900 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53032_pgum%2Fm",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53032/pgum/m"}
[junit4:junit4] 2> 890876 T3900 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52727_pgum%2Fm",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52727/pgum/m"}
[junit4:junit4] 2> 890876 T3900 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 890876 T3900 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 890879 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 890879 T3935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 890879 T3951 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 890879 T3906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 890879 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 891563 T3952 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372554440622/collection1
[junit4:junit4] 2> 891563 T3952 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 891564 T3952 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 891564 T3952 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 891565 T3952 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372554440622/collection1/'
[junit4:junit4] 2> 891566 T3952 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372554440622/collection1/lib/README' to classloader
[junit4:junit4] 2> 891567 T3952 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372554440622/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 891613 T3952 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 891663 T3952 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 891665 T3952 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 891668 T3952 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 892058 T3952 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 892058 T3952 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 892059 T3952 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 892065 T3952 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 892067 T3952 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 892077 T3952 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 892080 T3952 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 892082 T3952 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 892083 T3952 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 892083 T3952 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 892083 T3952 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 892084 T3952 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 892084 T3952 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 892084 T3952 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 892085 T3952 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1372554440622/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty3/
[junit4:junit4] 2> 892085 T3952 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@9261c5
[junit4:junit4] 2> 892085 T3952 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 892085 T3952 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty3
[junit4:junit4] 2> 892086 T3952 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty3/index/
[junit4:junit4] 2> 892086 T3952 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 892086 T3952 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty3/index
[junit4:junit4] 2> 892089 T3952 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 892090 T3952 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 892091 T3952 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 892092 T3952 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 892092 T3952 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 892093 T3952 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 892093 T3952 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 892094 T3952 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 892094 T3952 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 892094 T3952 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 892095 T3952 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 892096 T3952 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 892100 T3952 oass.SolrIndexSearcher.<init> Opening Searcher@1729d69 main
[junit4:junit4] 2> 892100 T3952 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 892100 T3952 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 892105 T3953 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1729d69 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 892107 T3952 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 892108 T3952 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52727/pgum/m collection:collection1 shard:shard1
[junit4:junit4] 2> 892110 T3952 oasc.ZkController.register We are http://127.0.0.1:52727/pgum/m/collection1/ and leader is http://127.0.0.1:55954/pgum/m/collection1/
[junit4:junit4] 2> 892110 T3952 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52727/pgum/m
[junit4:junit4] 2> 892110 T3952 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 892111 T3952 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1119 name=collection1 org.apache.solr.core.SolrCore@27a20a url=http://127.0.0.1:52727/pgum/m/collection1 node=127.0.0.1:52727_pgum%2Fm C1119_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:52727_pgum%2Fm, base_url=http://127.0.0.1:52727/pgum/m}
[junit4:junit4] 2> 892111 T3954 C1119 P52727 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 892111 T3952 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 892112 T3954 C1119 P52727 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 892112 T3954 C1119 P52727 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 892113 T3954 C1119 P52727 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 892113 T3877 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 892113 T3877 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 892114 T3954 C1119 P52727 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 892114 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 892119 T3915 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 892190 T3877 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 892191 T3877 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33740
[junit4:junit4] 2> 892192 T3877 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 892192 T3877 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 892193 T3877 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372554443326
[junit4:junit4] 2> 892193 T3877 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372554443326/solr.xml
[junit4:junit4] 2> 892193 T3877 oasc.CoreContainer.<init> New CoreContainer 32802100
[junit4:junit4] 2> 892194 T3877 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372554443326/'
[junit4:junit4] 2> 892194 T3877 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372554443326/'
[junit4:junit4] 2> 892248 T3877 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 892248 T3877 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 892249 T3877 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 892249 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 892249 T3877 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 892250 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 892250 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 892250 T3877 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 892251 T3877 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 892251 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 892254 T3877 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 892255 T3877 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42882/solr
[junit4:junit4] 2> 892255 T3877 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 892256 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 892257 T3966 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a1afc8 name:ZooKeeperConnection Watcher:127.0.0.1:42882 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 892258 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 892259 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 892262 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 892263 T3968 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e219ff name:ZooKeeperConnection Watcher:127.0.0.1:42882/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 892263 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 892265 T3877 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 892384 T3900 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 892385 T3900 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52727_pgum%2Fm",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52727/pgum/m"}
[junit4:junit4] 2> 892387 T3968 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 892387 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 892387 T3935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 892387 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 892387 T3951 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 892387 T3906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 893120 T3915 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 893120 T3915 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=4&state=recovering&nodeName=127.0.0.1:52727_pgum%252Fm&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 893268 T3877 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33740_pgum%2Fm
[junit4:junit4] 2> 893269 T3877 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33740_pgum%2Fm
[junit4:junit4] 2> 893271 T3968 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 893271 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 893271 T3906 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 893272 T3935 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 893271 T3951 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 893271 T3919 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 893272 T3951 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 893273 T3899 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 893272 T3935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 893272 T3906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 893273 T3968 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 893272 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 893278 T3969 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 893278 T3969 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 893891 T3900 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 893892 T3900 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33740_pgum%2Fm",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33740/pgum/m"}
[junit4:junit4] 2> 893892 T3900 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 893892 T3900 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 893896 T3968 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 893896 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 893896 T3935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 893896 T3951 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 893896 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 893896 T3906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 894279 T3969 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372554443326/collection1
[junit4:junit4] 2> 894279 T3969 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 894280 T3969 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 894280 T3969 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 894281 T3969 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372554443326/collection1/'
[junit4:junit4] 2> 894282 T3969 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372554443326/collection1/lib/README' to classloader
[junit4:junit4] 2> 894282 T3969 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372554443326/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 894313 T3969 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 894350 T3969 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 894352 T3969 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 894355 T3969 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 894754 T3969 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 894754 T3969 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 894755 T3969 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 894761 T3969 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 894764 T3969 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 894774 T3969 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 894777 T3969 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 894779 T3969 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 894780 T3969 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 894780 T3969 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 894780 T3969 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 894781 T3969 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 894781 T3969 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 894782 T3969 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 894782 T3969 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1372554443326/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty4/
[junit4:junit4] 2> 894782 T3969 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@9261c5
[junit4:junit4] 2> 894782 T3969 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 894783 T3969 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty4
[junit4:junit4] 2> 894783 T3969 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty4/index/
[junit4:junit4] 2> 894783 T3969 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 894783 T3969 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty4/index
[junit4:junit4] 2> 894786 T3969 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 894787 T3969 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 894788 T3969 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 894789 T3969 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 894789 T3969 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 894790 T3969 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 894790 T3969 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 894790 T3969 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 894791 T3969 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 894791 T3969 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 894792 T3969 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 894793 T3969 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 894796 T3969 oass.SolrIndexSearcher.<init> Opening Searcher@13f7fd0 main
[junit4:junit4] 2> 894796 T3969 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 894797 T3969 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 894802 T3970 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13f7fd0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 894804 T3969 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 894804 T3969 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33740/pgum/m collection:collection1 shard:shard2
[junit4:junit4] 2> 894806 T3969 oasc.ZkController.register We are http://127.0.0.1:33740/pgum/m/collection1/ and leader is http://127.0.0.1:53032/pgum/m/collection1/
[junit4:junit4] 2> 894806 T3969 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33740/pgum/m
[junit4:junit4] 2> 894806 T3969 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 894807 T3969 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1120 name=collection1 org.apache.solr.core.SolrCore@185ede1 url=http://127.0.0.1:33740/pgum/m/collection1 node=127.0.0.1:33740_pgum%2Fm C1120_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:33740_pgum%2Fm, base_url=http://127.0.0.1:33740/pgum/m}
[junit4:junit4] 2> 894807 T3971 C1120 P33740 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 894807 T3969 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 894808 T3971 C1120 P33740 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 894808 T3971 C1120 P33740 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 894808 T3971 C1120 P33740 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 894809 T3877 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 894809 T3877 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 894809 T3971 C1120 P33740 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 894809 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 894813 T3929 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 894883 T3877 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 894885 T3877 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56900
[junit4:junit4] 2> 894885 T3877 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 894886 T3877 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 894886 T3877 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372554446020
[junit4:junit4] 2> 894886 T3877 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372554446020/solr.xml
[junit4:junit4] 2> 894887 T3877 oasc.CoreContainer.<init> New CoreContainer 20582012
[junit4:junit4] 2> 894887 T3877 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372554446020/'
[junit4:junit4] 2> 894888 T3877 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372554446020/'
[junit4:junit4] 2> 894939 T3877 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 894940 T3877 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 894940 T3877 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 894941 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 894941 T3877 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 894941 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 894942 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 894942 T3877 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 894942 T3877 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 894943 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 894946 T3877 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 894946 T3877 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42882/solr
[junit4:junit4] 2> 894946 T3877 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 894947 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 894948 T3983 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65f821 name:ZooKeeperConnection Watcher:127.0.0.1:42882 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 894949 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 894950 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 894953 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 894954 T3985 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@482445 name:ZooKeeperConnection Watcher:127.0.0.1:42882/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 894955 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 894958 T3877 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C1121 name=collection1 org.apache.solr.core.SolrCore@27a20a url=http://127.0.0.1:52727/pgum/m/collection1 node=127.0.0.1:52727_pgum%2Fm C1121_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:52727_pgum%2Fm, base_url=http://127.0.0.1:52727/pgum/m}
[junit4:junit4] 2> 895121 T3954 C1121 P52727 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:55954/pgum/m/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 895122 T3954 C1121 P52727 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:52727/pgum/m START replicas=[http://127.0.0.1:55954/pgum/m/collection1/] nUpdates=100
[junit4:junit4] 2> 895122 T3954 C1121 P52727 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 895122 T3954 C1121 P52727 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 895122 T3954 C1121 P52727 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 895123 T3954 C1121 P52727 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 895123 T3954 C1121 P52727 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 895123 T3954 C1121 P52727 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:55954/pgum/m/collection1/. core=collection1
[junit4:junit4] 2> 895123 T3954 C1121 P52727 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1122 name=collection1 org.apache.solr.core.SolrCore@13d0898 url=http://127.0.0.1:55954/pgum/m/collection1 node=127.0.0.1:55954_pgum%2Fm C1122_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:55954_pgum%2Fm, base_url=http://127.0.0.1:55954/pgum/m, leader=true}
[junit4:junit4] 2> 895129 T3914 C1122 P55954 oasc.SolrCore.execute [collection1] webapp=/pgum/m path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 895134 T3912 C1122 P55954 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 895137 T3912 C1122 P55954 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 895137 T3912 C1122 P55954 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 895138 T3912 C1122 P55954 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 895138 T3912 C1122 P55954 oass.SolrIndexSearcher.<init> Opening Searcher@677364 realtime
[junit4:junit4] 2> 895139 T3912 C1122 P55954 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 895139 T3912 C1122 P55954 oasup.LogUpdateProcessor.finish [collection1] webapp=/pgum/m path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 895140 T3954 C1121 P52727 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 895140 T3954 C1121 P52727 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 895144 T3915 C1122 P55954 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 895145 T3915 C1122 P55954 oasc.SolrCore.execute [collection1] webapp=/pgum/m path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=4
[junit4:junit4] 2> 895145 T3954 C1121 P52727 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 895146 T3954 C1121 P52727 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 895146 T3954 C1121 P52727 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 895146 T3954 C1121 P52727 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 895147 T3954 C1121 P52727 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 895401 T3900 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 895402 T3900 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33740_pgum%2Fm",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33740/pgum/m"}
[junit4:junit4] 2> 895406 T3900 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52727_pgum%2Fm",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52727/pgum/m"}
[junit4:junit4] 2> 895412 T3906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 895413 T3951 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 895413 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 895413 T3985 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 895413 T3935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 895413 T3968 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 895413 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 895814 T3929 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 895814 T3929 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:33740_pgum%252Fm&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 895960 T3877 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56900_pgum%2Fm
[junit4:junit4] 2> 895961 T3877 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56900_pgum%2Fm
[junit4:junit4] 2> 895963 T3968 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 895963 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 895963 T3906 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 895963 T3935 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 895963 T3906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 895963 T3951 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 895963 T3985 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 895963 T3919 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 895964 T3985 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 895965 T3899 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 895964 T3951 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 895964 T3935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 895965 T3968 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 895965 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 895969 T3987 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 895970 T3987 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 896918 T3900 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 896919 T3900 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56900_pgum%2Fm",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56900/pgum/m"}
[junit4:junit4] 2> 896919 T3900 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 896919 T3900 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 896922 T3906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 896923 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 896923 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 896923 T3968 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 896922 T3951 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 896922 T3985 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 896922 T3935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 896971 T3987 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372554446020/collection1
[junit4:junit4] 2> 896971 T3987 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 896972 T3987 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 896972 T3987 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 896973 T3987 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372554446020/collection1/'
[junit4:junit4] 2> 896974 T3987 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372554446020/collection1/lib/README' to classloader
[junit4:junit4] 2> 896974 T3987 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372554446020/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 897010 T3987 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 897049 T3987 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 897050 T3987 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 897054 T3987 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 897440 T3987 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 897441 T3987 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 897441 T3987 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 897447 T3987 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 897450 T3987 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 897459 T3987 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 897461 T3987 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 897464 T3987 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 897465 T3987 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 897465 T3987 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 897465 T3987 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 897466 T3987 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 897466 T3987 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 897466 T3987 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 897466 T3987 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty5-1372554446020/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty5/
[junit4:junit4] 2> 897467 T3987 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@9261c5
[junit4:junit4] 2> 897467 T3987 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 897467 T3987 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty5
[junit4:junit4] 2> 897468 T3987 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty5/index/
[junit4:junit4] 2> 897468 T3987 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 897468 T3987 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty5/index
[junit4:junit4] 2> 897471 T3987 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 897471 T3987 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 897473 T3987 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 897473 T3987 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 897474 T3987 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 897474 T3987 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 897475 T3987 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 897475 T3987 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 897475 T3987 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 897476 T3987 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 897476 T3987 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 897478 T3987 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 897481 T3987 oass.SolrIndexSearcher.<init> Opening Searcher@274d02 main
[junit4:junit4] 2> 897481 T3987 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 897481 T3987 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 897493 T3988 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@274d02 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 897496 T3987 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 897496 T3987 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56900/pgum/m collection:collection1 shard:shard1
[junit4:junit4] 2> 897499 T3987 oasc.ZkController.register We are http://127.0.0.1:56900/pgum/m/collection1/ and leader is http://127.0.0.1:55954/pgum/m/collection1/
[junit4:junit4] 2> 897499 T3987 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56900/pgum/m
[junit4:junit4] 2> 897499 T3987 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 897499 T3987 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1123 name=collection1 org.apache.solr.core.SolrCore@be4656 url=http://127.0.0.1:56900/pgum/m/collection1 node=127.0.0.1:56900_pgum%2Fm C1123_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:56900_pgum%2Fm, base_url=http://127.0.0.1:56900/pgum/m}
[junit4:junit4] 2> 897500 T3989 C1123 P56900 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 897500 T3987 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 897501 T3989 C1123 P56900 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 897501 T3989 C1123 P56900 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 897501 T3989 C1123 P56900 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 897501 T3877 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 897502 T3989 C1123 P56900 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 897502 T3877 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 897503 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 897507 T3914 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 897577 T3877 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 897579 T3877 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52108
[junit4:junit4] 2> 897580 T3877 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 897580 T3877 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 897580 T3877 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372554448714
[junit4:junit4] 2> 897581 T3877 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372554448714/solr.xml
[junit4:junit4] 2> 897581 T3877 oasc.CoreContainer.<init> New CoreContainer 9937709
[junit4:junit4] 2> 897581 T3877 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372554448714/'
[junit4:junit4] 2> 897582 T3877 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372554448714/'
[junit4:junit4] 2> 897630 T3877 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 897630 T3877 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 897631 T3877 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 897631 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 897631 T3877 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 897632 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 897632 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 897632 T3877 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 897632 T3877 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 897633 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 897636 T3877 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 897636 T3877 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42882/solr
[junit4:junit4] 2> 897637 T3877 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 897637 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 897639 T4001 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13f1bf6 name:ZooKeeperConnection Watcher:127.0.0.1:42882 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 897639 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 897640 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 897643 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 897644 T4003 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12492c name:ZooKeeperConnection Watcher:127.0.0.1:42882/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 897644 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 897646 T3877 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C1124 name=collection1 org.apache.solr.core.SolrCore@185ede1 url=http://127.0.0.1:33740/pgum/m/collection1 node=127.0.0.1:33740_pgum%2Fm C1124_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:33740_pgum%2Fm, base_url=http://127.0.0.1:33740/pgum/m}
[junit4:junit4] 2> 897815 T3971 C1124 P33740 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:53032/pgum/m/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 897815 T3971 C1124 P33740 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:33740/pgum/m START replicas=[http://127.0.0.1:53032/pgum/m/collection1/] nUpdates=100
[junit4:junit4] 2> 897816 T3971 C1124 P33740 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 897816 T3971 C1124 P33740 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 897816 T3971 C1124 P33740 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 897816 T3971 C1124 P33740 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 897816 T3971 C1124 P33740 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 897817 T3971 C1124 P33740 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:53032/pgum/m/collection1/. core=collection1
[junit4:junit4] 2> 897817 T3971 C1124 P33740 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1125 name=collection1 org.apache.solr.core.SolrCore@1895779 url=http://127.0.0.1:53032/pgum/m/collection1 node=127.0.0.1:53032_pgum%2Fm C1125_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53032_pgum%2Fm, base_url=http://127.0.0.1:53032/pgum/m, leader=true}
[junit4:junit4] 2> 897822 T3929 C1125 P53032 oasc.SolrCore.execute [collection1] webapp=/pgum/m path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 897824 T3930 C1125 P53032 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 897826 T3930 C1125 P53032 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 897826 T3930 C1125 P53032 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 897827 T3930 C1125 P53032 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 897827 T3930 C1125 P53032 oass.SolrIndexSearcher.<init> Opening Searcher@1005cc4 realtime
[junit4:junit4] 2> 897827 T3930 C1125 P53032 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 897828 T3930 C1125 P53032 oasup.LogUpdateProcessor.finish [collection1] webapp=/pgum/m path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4] 2> 897828 T3971 C1124 P33740 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 897828 T3971 C1124 P33740 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 897830 T3928 C1125 P53032 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 897830 T3928 C1125 P53032 oasc.SolrCore.execute [collection1] webapp=/pgum/m path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 897830 T3971 C1124 P33740 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 897830 T3971 C1124 P33740 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 897831 T3971 C1124 P33740 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 897831 T3971 C1124 P33740 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 897831 T3971 C1124 P33740 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 898426 T3900 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 898427 T3900 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56900_pgum%2Fm",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56900/pgum/m"}
[junit4:junit4] 2> 898429 T3900 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33740_pgum%2Fm",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33740/pgum/m"}
[junit4:junit4] 2> 898432 T3906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 898432 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 898432 T4003 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 898432 T3968 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 898432 T3985 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 898432 T3935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 898432 T3951 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 898432 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 898508 T3914 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 898508 T3914 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=6&state=recovering&nodeName=127.0.0.1:56900_pgum%252Fm&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 898648 T3877 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52108_pgum%2Fm
[junit4:junit4] 2> 898651 T3877 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52108_pgum%2Fm
[junit4:junit4] 2> 898652 T3968 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 898652 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 898653 T3985 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 898653 T3985 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 898653 T3935 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 898654 T3935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 898653 T3906 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 898653 T3951 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 898653 T3919 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 898654 T3968 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 898654 T3951 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 898654 T3906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 898655 T4003 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 898655 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 898656 T3899 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 898656 T4003 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 898660 T4005 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 898660 T4005 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 899938 T3900 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 899939 T3900 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52108_pgum%2Fm",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52108/pgum/m"}
[junit4:junit4] 2> 899939 T3900 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4] 2> 899939 T3900 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 899942 T3906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 899942 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 899942 T3951 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 899942 T3935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 899942 T3968 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 899942 T4003 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 899942 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 899942 T3985 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> C1123_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:56900_pgum%2Fm, base_url=http://127.0.0.1:56900/pgum/m}
[junit4:junit4] 2> 900510 T3989 C1123 P56900 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:55954/pgum/m/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 900510 T3989 C1123 P56900 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56900/pgum/m START replicas=[http://127.0.0.1:55954/pgum/m/collection1/] nUpdates=100
[junit4:junit4] 2> 900510 T3989 C1123 P56900 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 900511 T3989 C1123 P56900 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 900511 T3989 C1123 P56900 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 900511 T3989 C1123 P56900 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 900511 T3989 C1123 P56900 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 900511 T3989 C1123 P56900 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:55954/pgum/m/collection1/. core=collection1
[junit4:junit4] 2> 900511 T3989 C1123 P56900 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1126 name=collection1 org.apache.solr.core.SolrCore@13d0898 url=http://127.0.0.1:55954/pgum/m/collection1 node=127.0.0.1:55954_pgum%2Fm C1126_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:55954_pgum%2Fm, base_url=http://127.0.0.1:55954/pgum/m, leader=true}
[junit4:junit4] 2> 900512 T3914 C1126 P55954 oasc.SolrCore.execute [collection1] webapp=/pgum/m path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 900514 T3915 C1126 P55954 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 900515 T3915 C1126 P55954 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 900515 T3915 C1126 P55954 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 900515 T3915 C1126 P55954 oasup.LogUpdateProcessor.finish [collection1] webapp=/pgum/m path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 1
[junit4:junit4] 2> 900516 T3989 C1123 P56900 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 900516 T3989 C1123 P56900 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 900518 T3913 C1126 P55954 oasc.SolrCore.execute [collection1] webapp=/pgum/m path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 900518 T3989 C1123 P56900 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 900519 T3989 C1123 P56900 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 900519 T3989 C1123 P56900 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 900519 T3989 C1123 P56900 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 900520 T3989 C1123 P56900 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 900661 T4005 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372554448714/collection1
[junit4:junit4] 2> 900662 T4005 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 900662 T4005 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 900662 T4005 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 900663 T4005 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372554448714/collection1/'
[junit4:junit4] 2> 900664 T4005 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372554448714/collection1/lib/README' to classloader
[junit4:junit4] 2> 900664 T4005 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372554448714/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 900693 T4005 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 900729 T4005 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 900730 T4005 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 900734 T4005 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 901111 T4005 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 901111 T4005 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 901112 T4005 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 901118 T4005 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 901120 T4005 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 901129 T4005 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 901131 T4005 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 901134 T4005 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 901135 T4005 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 901135 T4005 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 901135 T4005 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 901136 T4005 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 901136 T4005 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 901137 T4005 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 901137 T4005 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty6-1372554448714/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty6/
[junit4:junit4] 2> 901137 T4005 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@9261c5
[junit4:junit4] 2> 901137 T4005 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 901138 T4005 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty6
[junit4:junit4] 2> 901138 T4005 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty6/index/
[junit4:junit4] 2> 901138 T4005 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 901139 T4005 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty6/index
[junit4:junit4] 2> 901141 T4005 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1372554431074/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 901142 T4005 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 901143 T4005 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 901143 T4005 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 901144 T4005 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 901145 T4005 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 901145 T4005 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 901145 T4005 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 901147 T4005 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 901147 T4005 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 901148 T4005 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 901149 T4005 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 901152 T4005 oass.SolrIndexSearcher.<init> Opening Searcher@f72b51 main
[junit4:junit4] 2> 901153 T4005 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 901153 T4005 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 901158 T4007 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f72b51 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 901160 T4005 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 901160 T4005 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52108/pgum/m collection:collection1 shard:shard2
[junit4:junit4] 2> 901163 T4005 oasc.ZkController.register We are http://127.0.0.1:52108/pgum/m/collection1/ and leader is http://127.0.0.1:53032/pgum/m/collection1/
[junit4:junit4] 2> 901163 T4005 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52108/pgum/m
[junit4:junit4] 2> 901163 T4005 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 901163 T4005 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1127 name=collection1 org.apache.solr.core.SolrCore@d59f70 url=http://127.0.0.1:52108/pgum/m/collection1 node=127.0.0.1:52108_pgum%2Fm C1127_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:52108_pgum%2Fm, base_url=http://127.0.0.1:52108/pgum/m}
[junit4:junit4] 2> 901163 T4008 C1127 P52108 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 901164 T4005 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 901164 T4008 C1127 P52108 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 901164 T4008 C1127 P52108 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 901165 T4008 C1127 P52108 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 901165 T3877 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 901165 T4008 C1127 P52108 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 901165 T3877 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 901166 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 901169 T3931 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 901235 T3877 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 901237 T3877 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49485
[junit4:junit4] 2> 901238 T3877 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 901238 T3877 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 901238 T3877 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty7-1372554452376
[junit4:junit4] 2> 901239 T3877 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty7-1372554452376/solr.xml
[junit4:junit4] 2> 901239 T3877 oasc.CoreContainer.<init> New CoreContainer 2524163
[junit4:junit4] 2> 901239 T3877 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty7-1372554452376/'
[junit4:junit4] 2> 901240 T3877 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty7-1372554452376/'
[junit4:junit4] 2> 901292 T3877 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 901293 T3877 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 901293 T3877 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 901294 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 901294 T3877 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 901294 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 901294 T3877 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 901295 T3877 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 901295 T3877 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 901295 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 901298 T3877 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 901299 T3877 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42882/solr
[junit4:junit4] 2> 901299 T3877 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 901300 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 901301 T4020 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@edf683 name:ZooKeeperConnection Watcher:127.0.0.1:42882 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 901302 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 901303 T3877 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 901306 T3877 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 901307 T4022 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1056657 name:ZooKeeperConnection Watcher:127.0.0.1:42882/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 901307 T3877 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 901310 T3877 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 901446 T3900 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 901447 T3900 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56900_pgum%2Fm",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56900/pgum/m"}
[junit4:junit4] 2> 901450 T3900 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "numShards":"2",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52108_pgum%2Fm",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52108/pgum/m"}
[junit4:junit4] 2> 901454 T4022 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 901455 T3899 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 901454 T3968 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 901454 T3951 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 901454 T3935 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 901454 T4003 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 901454 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 901454 T3985 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 901454 T3906 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 902169 T3931 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 902170 T3931 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=7&state=recovering&nodeName=127.0.0.1:52108_pgum%252Fm&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 902312 T3877 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49485_pgum%2Fm
[junit4:junit4] 2> 902314 T3877 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49485_pgum%2Fm
[junit4:junit4] 2> 902316 T3968 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 902316 T3919 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 902316 T3919 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 902316 T3935 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 902317 T3951 oascc.ZkStat
[...truncated too long message...]
Pool.java:608)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] > 5) Thread[id=4078, name=HashSessionScavenger-123, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4:junit4] > at java.lang.Object.wait(Native Method)
[junit4:junit4] > at java.util.TimerThread.mainLoop(Timer.java:552)
[junit4:junit4] > at java.util.TimerThread.run(Timer.java:505)
[junit4:junit4] > 6) Thread[id=4082, name=qtp6705546-4082 Acceptor1 SelectChannelConnector@127.0.0.1:53032, state=BLOCKED, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4:junit4] > at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
[junit4:junit4] > at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
[junit4:junit4] > at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] > 7) Thread[id=4083, name=qtp6705546-4083, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4:junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4:junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2077)
[junit4:junit4] > at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] > 8) Thread[id=4084, name=qtp6705546-4084, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4:junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4:junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2077)
[junit4:junit4] > at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] > 9) Thread[id=4079, name=qtp6705546-4079 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4:junit4] > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[junit4:junit4] > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:257)
[junit4:junit4] > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
[junit4:junit4] > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4:junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
[junit4:junit4] > at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
[junit4:junit4] > at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([4F95D3036EBA0AEA]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4] > 1) Thread[id=4081, name=qtp6705546-4081 Acceptor0 SelectChannelConnector@127.0.0.1:53032, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4:junit4] > at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)
[junit4:junit4] > at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] > 2) Thread[id=4080, name=qtp6705546-4080 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4:junit4] > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[junit4:junit4] > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:257)
[junit4:junit4] > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
[junit4:junit4] > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4:junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
[junit4:junit4] > at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
[junit4:junit4] > at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] > 3) Thread[id=4078, name=HashSessionScavenger-123, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4:junit4] > at java.lang.Object.wait(Native Method)
[junit4:junit4] > at java.util.TimerThread.mainLoop(Timer.java:552)
[junit4:junit4] > at java.util.TimerThread.run(Timer.java:505)
[junit4:junit4] > 4) Thread[id=4082, name=qtp6705546-4082 Acceptor1 SelectChannelConnector@127.0.0.1:53032, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4:junit4] > at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)
[junit4:junit4] > at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] > 5) Thread[id=4079, name=qtp6705546-4079 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeyNothingIsSafeTest]
[junit4:junit4] > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[junit4:junit4] > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:257)
[junit4:junit4] > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
[junit4:junit4] > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4:junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
[junit4:junit4] > at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
[junit4:junit4] > at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([4F95D3036EBA0AEA]:0)
[junit4:junit4] Completed on J0 in 112.41s, 1 test, 2 errors <<< FAILURES!
[...truncated 71 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1248: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:892: There were test failures: 299 suites, 1290 tests, 2 suite-level errors, 99 ignored (7 assumptions)
Total time: 35 minutes 41 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-ea-b94 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure